Labs 02: Lexer for Tool
This assignment is the first stage of the Amy compiler.
To merge the stubs for the Lexer into your current branch (which should contain a working Interpreter by now), use
git fetch origin git merge origin/Lab02
- We are no longer going to use the parser+trees given in
lib/amyc-frontend_2.12-1.0.jar. This commit removes this file.
Frontendis no longer available,
InterpreterTestsdo not make sense so they are removed. This can create conflicts with your assignment if you changed them. If so, do
git rm test/scala/amyc/test/InterpreterTests.scala git rm test/resources/interpreter git commit
The structure of your
src directory should be as follows:
amyc ├── Main.scala (updated) │ ├── analyzer (new, for later) │ └── SymbolTable.scala │ ├── ast (new, for later) │ ├── Identifier.scala │ ├── Printer.scala │ └── TreeModule.scala │ ├── interpreter │ └── Interpreter.scala │ ├── parsing (new) │ ├── Lexer.scala │ └── Tokens.scala │ └── utils (new) ├── AmycFatalError.scala ├── Context.scala ├── Document.scala ├── Pipeline.scala ├── Position.scala ├── Reporter.scala └── UniqueCounter.scala
analyzer directories will be used later, but are included as dependencies of the interpreter.
This lab will focus on the following two files:
parsing/Tokens.scala: list of tokens and token classes.
parsing/Lexer.scala: stub for the
A Lexer for Amy
The role of a lexer is to read the input text and convert it to a list of tokens. Tokens are the smallest useful units in a source file: a name referring to a variable, a bracket, a keyword etc. The role of the lexer is to group together those useful units (e.g. return the keyword
else as a unit, as opposed to individual characters
e) and to abstract away all useless information (i.e. whitespace, comments).
You can find the lexer in the
Lexer.scala file. Its implementation is based on Streams, which are lazy sequences of elements, i.e. their elements are not produced all at the creation of the collection, but by demand. Unlike iterators, they are immutable, i.e. elements are not consumed when they are created, but remain in the Stream. You can find the Scala Stream API here.
The Lexer has the following components:
- The public method is
run. It just calls
lexFilefor every input file and concatenates the results.
lexFilewill lex one input file. It takes as input a Stream of pairs of
(Char, Position)(which we generate for you from the input via the
tokenStream, which in turn calls
nextTokenuntil the input is empty.
nextTokenis essentially a wrapper method around
readToken. It removes whitespace and comments and then calls
readToken. This recursive method has to be tail-recursive so it does not stack overflow on long inputs (this is imposed with
readTokenis the main function of the lexer. It will look at the current character and decide what kind of token it has to emit, possibly reading more characters along the way. As said, your input is an immutable Stream so you don't have to worry about consuming characters, you can peak at them at will. Look at the utilities at the beginning of
- Finally, the
keywordsmethod returns a keyword token corresponding to a string, or
Noneif there is not corresponding keyword.
It is your task to complete
Here are some details you should pay attention to:
- Make sure you recognize keywords as their own token type.
else, for instance, should be lexed as a token
ELSE(), not as an identifier with the content “while”.
- Make sure you correctly register the position of all tokens. Note the
currentPoson the head of the input stream and the utility
- In general, it is good to output as many errors as possible (this helps whoever uses your compiler). This means your lexer should not give up after the first error, but rather skip the erroneous token, emit an error message, and then continue lexing. Use the
BADtoken to mark erroneous tokens. However, make sure to also call the
ctx.reporter: this will register the error and eventually fail your program.
- The Lexer does not immediately read and return all tokens, it returns an Stream[Token] that will be used by future phases to read tokens on demand.
- Comments should not produce tokens.
- Returned tokens should be fresh instances of the the appropriate Token subclass. Value tokens (tokens that carry a value, such as identifiers), need to be constructed with the appropriate value.
- You should apply the longest matching rule:
- Make sure to correctly implement the Amy lexing rules for literals and identifiers.
Your compiler needs to exit with a non-0 code in case it encounters an error. Bad tokens of invalid characters are considered as such errors.
Reporter available in the
ctx parameter of the
run method, provides a method
error which will store in the Reporter the information that an error has occurred.
Reporter also provides the method
terminateIfErrors which will terminate the compiler if an error has occurred before. This method is already called between each phase of the compiler. To make sure that the compiler also terminates with non-0 code in case the *last* phase emits an error, it is also called in the end of
(Notice that, since your lexer phase returns an Iterator, errors might be emitted by your lexer while consuming the iterator during the DisplayToken phase)
In short, make sure you call
error whenever you encounter an erroneous character, to make sure that your compiler will eventually exit with an error code. For the lexing phase, avoid
fatal as this will terminate the compiler immediately (and not allow you to report multiple errors).
For reference, here is a possible output for the example under
examples/Hello.scala. You can always get reference output for the lexer from the reference compiler by typing
java -jar amyc_2.12-1.X.jar --printTokens <files>
OBJECT()(1:1) ID(Hello)(1:8) EXTENDS()(1:14) APP()(1:22) LBRACE()(1:26) ID(Std)(2:3) DOT()(2:6) ID(printString)(2:7) LPAREN()(2:18) STRINGLIT(Hello )(2:19) CONCAT()(2:28) STRINGLIT(world!)(2:31) RPAREN()(2:39) RBRACE()(3:1) EOF()(3:2)
You are given 2 weeks for this assignment.
Deadline: Tuesday, Oct. 10, 23.59pm.
You must use the interface on https://larasrv13.epfl.ch/clp17 as follows:
- Click on the “Deliverables” tab
- Click “Deliver this version” on the commit corresponding to the version of your code you want to submit