Package com.javacc.parser

Class Token

java.lang.Object

com.javacc.parser.Token

All Implemented Interfaces:

JavaCCConstants, Node, freemarker.template.TemplateModel, freemarker.template.TemplateNodeModel, freemarker.template.TemplateScalarModel, Comparable<Node>

Direct Known Subclasses:

ANY_CHAR, Backslash, Comment, Delimiter, HASH, Identifier, InvalidToken, JavaCCKeyWord, KeyWord, Literal, MULTI_LINE_COMMENT_START, Operator, RightArrow, SINGLE_LINE_COMMENT_START, UP_TO_HERE, Whitespace

public class Token
extends Object
implements JavaCCConstants, Node

Nested Class Summary

Nested classes/interfaces inherited from interface com.javacc.parser.JavaCCConstants

JavaCCConstants.LexicalState, JavaCCConstants.TokenType

Nested classes/interfaces inherited from interface com.javacc.parser.Node

Node.Visitor

Nested classes/interfaces inherited from interface freemarker.template.TemplateModel

freemarker.template.TemplateModel.InvalidExpressionModel, freemarker.template.TemplateModel.JavaNull

Field Summary

Fields inherited from interface com.javacc.parser.JavaCCConstants

BUF_SIZE

Fields inherited from interface freemarker.template.TemplateModel

INVALID_EXPRESSION, JAVA_NULL, NOTHING

Fields inherited from interface freemarker.template.TemplateScalarModel

EMPTY_STRING

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	Token()
	Token(JavaCCConstants.TokenType type, JavaCCLexer tokenSource, int beginOffset, int endOffset)
	Token(JavaCCConstants.TokenType type, String image, JavaCCLexer tokenSource)

Method Summary

Modifier and Type	Method	Description
void	addChild(int i, Node n)	inserts a child Node at a specific index, displacing the nodes after the index by 1.
void	addChild(Node n)	appends a child node to this Node
List<Node>	children()
void	clearChildren()	Remove all the child nodes
void	copyLocationInfo(Node from)	Copy the location info from a Node
void	copyLocationInfo(Node start, Node end)	Copy the location info given a start and end Node
Iterator<Token>	followingTokens()
String	getAsString()
int	getBeginOffset()
Node	getChild(int i)
int	getChildCount()
freemarker.template.TemplateSequenceModel	getChildNodes()
int	getEndOffset()
Grammar	getGrammar()
String	getImage()
String	getLeadingComments()
String	getLocation()
Token	getNext()
String	getNodeName()
String	getNodeNamespace()
String	getNodeType()
String	getNormalizedText()
Node	getParent()
freemarker.template.TemplateNodeModel	getParentNode()
Token	getPrevious()
String	getSource()
JavaCCLexer	getTokenSource()
JavaCCConstants.TokenType	getType()	Return the TokenType of this Token object
int	indexOf(Node n)
boolean	isInserted()
boolean	isSkipped()
boolean	isUnparsed()
boolean	isVirtual()
static Token	newToken(JavaCCConstants.TokenType type, JavaCCLexer tokenSource, int beginOffset, int endOffset)
static Token	newToken(JavaCCConstants.TokenType type, String image, JavaCCLexer tokenSource)
Token	nextCachedToken()
Iterator<Token>	precedingTokens()
List<Token>	precedingUnparsedTokens()
void	preInsert(Token prependedToken)
Token	previousCachedToken()
Node	removeChild(int i)	Remove the node at index i.
Token	replaceType(JavaCCConstants.TokenType type)
void	setBeginOffset(int beginOffset)	It would be extremely rare that an application programmer would use this method.
void	setChild(int i, Node n)	Replace the node at index i
void	setEndOffset(int endOffset)	It would be extremely rare that an application programmer would use this method.
void	setGrammar(Grammar grammar)
void	setImage(String image)
void	setParent(Node parent)
void	setTokenSource(JavaCCLexer tokenSource)	It should be exceedingly rare that an application programmer needs to use this method.
protected void	setType(JavaCCConstants.TokenType type)
void	setUnparsed(boolean unparsed)	Mark whether this Node is unparsed, i.e.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.javacc.parser.Node

appendChild, children, childrenOfType, close, compareTo, descendants, descendants, descendants, descendants, descendantsOfType, dump, dump, firstAncestorOfType, firstChildOfType, firstChildOfType, firstChildOfType, firstDescendantOfType, firstDescendantOfType, getAllTokens, getBeginColumn, getBeginLine, getEndColumn, getEndLine, getFirstChild, getFirstToken, getInputSource, getLastChild, getLastToken, getRealTokens, getRoot, getTokenType, hasChildNodes, iterator, nextSibling, open, prependChild, previousSibling, removeChild, replace, replaceChild

Constructor Detail

Token

public Token(JavaCCConstants.TokenType type,
             String image,
             JavaCCLexer tokenSource)

Parameters:

type - the #TokenType of the token being constructed

image - the String content of the token

tokenSource - the object that vended this token.

Token

protected Token()

Token

public Token(JavaCCConstants.TokenType type,
             JavaCCLexer tokenSource,
             int beginOffset,
             int endOffset)

Method Detail

getGrammar

public Grammar getGrammar()

Specified by:

getGrammar in interface Node

setGrammar

public void setGrammar(Grammar grammar)

Specified by:

setGrammar in interface Node

getNodeName

public String getNodeName()

Specified by:

getNodeName in interface freemarker.template.TemplateNodeModel

getLeadingComments

public String getLeadingComments()

getNormalizedText

public String getNormalizedText()

setImage

public void setImage(String image)

isInserted

public boolean isInserted()

preInsert

public void preInsert(Token prependedToken)

newToken

public static Token newToken(JavaCCConstants.TokenType type,
                             String image,
                             JavaCCLexer tokenSource)

setBeginOffset

public void setBeginOffset(int beginOffset)

It would be extremely rare that an application programmer would use this method. It needs to be public because it is part of the com.javacc.parser.Node interface.

Specified by:

setBeginOffset in interface Node

setEndOffset

public void setEndOffset(int endOffset)

It would be extremely rare that an application programmer would use this method. It needs to be public because it is part of the com.javacc.parser.Node interface.

Specified by:

setEndOffset in interface Node

getTokenSource

public JavaCCLexer getTokenSource()

Specified by:

getTokenSource in interface Node

Returns:

the JavaCCLexer object that handles location info for the tokens.

setTokenSource

public void setTokenSource(JavaCCLexer tokenSource)

It should be exceedingly rare that an application programmer needs to use this method.

Specified by:

setTokenSource in interface Node

getType

public JavaCCConstants.TokenType getType()

Return the TokenType of this Token object

setType

protected void setType(JavaCCConstants.TokenType type)

isVirtual

public boolean isVirtual()

Returns:

whether this Token represent actual input or was it inserted somehow?

isSkipped

public boolean isSkipped()

Returns:

Did we skip this token in parsing?

getBeginOffset

public int getBeginOffset()

Specified by:

getBeginOffset in interface Node

Returns:

the offset in the input source where the token begins, expressed in code units.

getEndOffset

public int getEndOffset()

Specified by:

getEndOffset in interface Node

Returns:

the offset in the input source where the token ends, expressed in code units. This is actually the offset where the very next token would begin.

getImage

public String getImage()

Returns:

the string image of the token.

getNext

public final Token getNext()

Returns:

the next _cached_ regular (i.e. parsed) token or null

getPrevious

public final Token getPrevious()

Returns:

the previous regular (i.e. parsed) token or null

nextCachedToken

public Token nextCachedToken()

Returns:

the next token of any sort (parsed or unparsed or invalid)

previousCachedToken

public Token previousCachedToken()

replaceType

public Token replaceType(JavaCCConstants.TokenType type)

getSource

public String getSource()

Specified by:

getSource in interface Node

Returns:

the original source content this Node came from a reference to the #JavaCCLexer that stores the source code and the start/end location info stored in the Node object itself. This method could throw a NullPointerException if #getTokenSource returns null. Also, the return value could be spurious if the content of the source file was changed meanwhile. But this is just the default implementation of an API and it does not address this problem!

isUnparsed

public boolean isUnparsed()

Specified by:

isUnparsed in interface Node

Returns:

whether this Node was created by regular operations of the parsing machinery.

setUnparsed

public void setUnparsed(boolean unparsed)

Description copied from interface: Node

Mark whether this Node is unparsed, i.e. not the result of normal parsing

Specified by:

setUnparsed in interface Node

Parameters:

unparsed - whether to set the Node as unparsed or parsed.

clearChildren

public void clearChildren()

Description copied from interface: Node

Remove all the child nodes

Specified by:

clearChildren in interface Node

toString

public String toString()

Overrides:

toString in class Object

precedingTokens

public Iterator<Token> precedingTokens()

Returns:

An iterator of the tokens preceding this one.

precedingUnparsedTokens

public List<Token> precedingUnparsedTokens()

Returns:

a list of the unparsed tokens preceding this one in the order they appear in the input

followingTokens

public Iterator<Token> followingTokens()

Returns:

An iterator of the (cached) tokens that follow this one.

copyLocationInfo

public void copyLocationInfo(Node from)

Copy the location info from a Node

Specified by:

copyLocationInfo in interface Node

Parameters:

from - the Node to copy the info from

copyLocationInfo

public void copyLocationInfo(Node start,
                             Node end)

Description copied from interface: Node

Copy the location info given a start and end Node

Specified by:

copyLocationInfo in interface Node

Parameters:

start - the start node

end - the end node

newToken

public static Token newToken(JavaCCConstants.TokenType type,
                             JavaCCLexer tokenSource,
                             int beginOffset,
                             int endOffset)

getLocation

public String getLocation()

Specified by:

getLocation in interface Node

Returns:

a String that gives the starting location of this Node. This is a default implementation that could be overridden

setChild

public void setChild(int i,
                     Node n)

Description copied from interface: Node

Replace the node at index i

Specified by:

setChild in interface Node

Parameters:

i - the index

n - the node

addChild

public void addChild(Node n)

Description copied from interface: Node

appends a child node to this Node

Specified by:

addChild in interface Node

Parameters:

n - the Node to append

addChild

public void addChild(int i,
                     Node n)

Description copied from interface: Node

inserts a child Node at a specific index, displacing the nodes after the index by 1.

Specified by:

addChild in interface Node

Parameters:

i - the (zero-based) index at which to insert the node

n - the Node to insert

removeChild

public Node removeChild(int i)

Description copied from interface: Node

Remove the node at index i. Any Nodes after i are shifted to the left.

Specified by:

removeChild in interface Node

Parameters:

i - the index at which to remove

Returns:

the removed Node

indexOf

public final int indexOf(Node n)

Specified by:

indexOf in interface Node

Parameters:

n - the Node to get the index of

Returns:

the index of the child Node. Or -1 if it is not a child Node.

getParent

public Node getParent()

Specified by:

getParent in interface Node

Returns:

this node's parent Node

setParent

public void setParent(Node parent)

Specified by:

setParent in interface Node

Parameters:

parent - The Node to set as the parent. Mostly used internally. The various addChild or appendChild sorts of methods should use this to set the node's parent.

getChildCount

public final int getChildCount()

Specified by:

getChildCount in interface Node

Returns:

the number of child nodes

getChild

public final Node getChild(int i)

Specified by:

getChild in interface Node

Parameters:

i - the index of the Node to return

Returns:

the Node at the specific offset

children

public final List<Node> children()

Specified by:

children in interface Node

getParentNode

public freemarker.template.TemplateNodeModel getParentNode()

Specified by:

getParentNode in interface freemarker.template.TemplateNodeModel

getChildNodes

public freemarker.template.TemplateSequenceModel getChildNodes()

Specified by:

getChildNodes in interface freemarker.template.TemplateNodeModel

getNodeType

public String getNodeType()

Specified by:

getNodeType in interface freemarker.template.TemplateNodeModel

getNodeNamespace

public String getNodeNamespace()

Specified by:

getNodeNamespace in interface freemarker.template.TemplateNodeModel

getAsString

public String getAsString()

Specified by:

getAsString in interface freemarker.template.TemplateScalarModel