literal thoughts

perl6.vim gets more love

Sun, 01 Mar 2015 20:30:00 +0000

tl;dr: Some tuits came my way, so Vim’s syntax highlighting of Perl 6 is much better and faster now. Try it!

<timotimo> hoelzro: i just skimmed the perl6 vim syntax and … oh crap

<timotimo> there’s just no way i could even make a dent in that thing

* psch usually turns off highlighting in vim because it’s so slow :/

<sorear> perl 6 is not really syntax-highlightable

So…yeah, perl6.vim is a bit of a beast.

Recent history of perl6.vim

Sometime in 2008 or so, I got interested in Perl 6, and I really wanted it to be nicely higlighted in my editor (Vim) as I learned. As it turned out, there was already a Vim syntax file for it, in the Pugs SVN repository. This was mostly the work of lpalmer++ and moritz++. It worked decently, but as I learned about more nooks and crannies of Perl 6 (there sure are enough of them) I started finding the limitations of the highlighting to be frustrating.

I wanted to make it better, so I started learning all about Vim’s regexes and syntax highlighting support. I began hacking on perl6.vim, and soon enough it grew to be the largest Vim syntax file in existence. Perl 6 is hard enough to parse already. Having to do so with Vim patterns, doubly so. I made some progress but eventually the performance of the highlighting suffered as it grew to support ever more complicated syntax.

In 2009, alester++ created the vim-perl repository on Github to consolidat Perl support for Vim. perl6.vim gained a new home shortly thereafter when he imported it from the Pugs repo. Around the same I discovered Text::VimColor and used it to create a test harness for the syntax highlighting of perl.vim and perl6.vim. I fixed a few Perl 5 highlighting bugs and added some tests for them, but I never got around to doing the same for Perl 6.

Since then, a few contributors (hoelzro++ et al) have continued to fix bugs in perl6.vim. In the meantime, Perl 6 has also evolved, and some of the assumptions perl6.vim has made are no longer valid.

A few weeks ago I went to FOSDEM 2015, where Larry announced that this Christmas would be The One™. That rekindled my interest in perl6.vim, and I’d like to share with you some of the progress I’ve made with it.

Performance

Vim 7.4 ships with a profiler for its syntax highlighting. I used it to identify a lot of inefficient patterns in the syntax file. I tried to reduce the use of zero-width assertions (lookarounds) and to make more use of nextgroup instead when possible. I also optimized many patterns to reject matches as soon as possible. As a result, Vim now needs to attempt much fewer matches, and the ones it does attempt are taking less time to do their work. Editing most Perl 6 files feels pretty snappy now, even STD.pm6.

Identifiers

In ftplugin/perl6.vim, I added the apostrophe and high-bit alphabetical characters (æóð...) to the list of characters vim recognizes in keywords (iskeyword) This allows all valid Perl 6 identifiers to be used in Vim commands that depend on matching keywords.

I also cleaned up the highlighting of Perl 6 idenfifiers, in particular adding support for the high-bit characters and better support for dashes, so these are now highlighted correctly in sub/package/variable/etc names.

Multiline comments

These have been updated to recognize the new #` prefix, and to be allowed at the beginning of a line. Previously, multiline comments with stacked delimiters (««, <<, <<<, etc) were not highlighted by default. Some profiling showed that doing so does not have a noticable performance impact, so I changed it to always highlight them.

Heredocs

I added highlighting of heredocs with qto, qqto, q:to, qq:to, q:heredoc, and qq:heredoc, for most commonly used delimiters.

Setting functions and methods

* TimToady wishes all these highlighters wouldn’t treat setting functions and methods as reserved words; they’re just functions and methods`

<moritz> TimToady: agreed

* flussence wishes perl6.vim didn’t highlight Test.pm functions as reserved words either…

By popular request, I removed highlighting of builtin functions and methods. This makes sense since there’s no way to avoid highlighting user-defined methods that happen to share the name of a builtin one.

Metaoperators

Highlighting these is one of the trickiest jobs of the syntax file. Yet it’s one of the most crucial because if you let e.g. a stray R< or [/ go unhighlighted, they will screw up the highlighting of the rest of the file by starting a string where there isn’t one.

I managed to make great improvements to the highlighting of reduce ([+]), hyper (»+«), post-hyper (@foo».bar), reverse/cross/sequence/zip (Rdiv, R<, etc), and set operators ((<), (>=)). All the spec tests for metaoperators I have looked at are now highlighted without issues.

Strings and patterns

Another part of the language that presents a challenge is /foo/, <bar>, <<baz>>, and «quux». Determining when these are not actually numeric division, less-than, or a hyperoperators is tricky. These things are now matched much more accurately. I also added support for the qqx and qqw operators.

Grammars

Grammars are a whole other language in their own right, and this is where there is still the most room for improvement (of both performance and accuracy) in the syntax file. I still don’t know grammars well enough, but I’ve fixed all the highlighting issues I’ve found. STD.pm6 in its entirety is now highlighted properly, although Vim can still get confused occasionally until you scroll a bit or tell it to redraw the screen.

Pod

The biggest change here is that I added highlighting of indented Pod blocks. As part of the indentifier improvements mentioned above, Pod block names containing hyphens or high-bit alphabetical chars are now correctly highlighted.

Folding

I added rudimentary support for syntax-based folding (disabled by default).

Miscellaneous

I improved highlighting of numbers and version literals
Fixed a few issues where interpolated things weren’t highlighted as such
Made big improvements to highlighting of transliteration (tr///, tr{}{}, etc) operators
Added highlighting of bare/anonymous sigils in more places
Highlighted the Unicode set operators (∈, ≽, et al)
Certain keywords (like die, state, etc) are no longer highlighted where they shouldn’t be, e.g. as method calls ($foo.state)

Testing

Of course, I added tests for every improvement and bugfix listed above, so at least the highlighting should not be getting worse from now on.

END

All open vim-perl Github issues relating to Perl 6 highlighting bugs have been resolved. If you discover any more problems, please open new issues.

I tried patching perl6/doc to add an option to use Text::VimColor to highlight code blocks in the documentation. While it does result in richer and more accurate highlighting than the default pygments-based highlighter, it is painfully slow to generate. This is because Text::VimColor (and 2html.vim, on which it is based) extracts Vim’s highlighting by calling the synID() function at every character position in the file, which is very slow. So while it may only take Vim a few dozen milliseconds to highlight an entire file, extracting the highlighting state into something usable by an external tool takes several seconds. I wonder how difficult it would be to patch Vim (or NeoVim) to export an entire file’s highlighting in a more efficient manner…

To close, here is a sample of some meaningless code that used to get perl6.vim confused but just looks nice now:

class Johnny's::Super-Cool::Module;

my $cööl-páttérn #`[why hello there] = /foo+/;
sub infix:«foo» { }

sub foo-bar (Int $a where * > 3, @bar, Str $b = 'foo') {
    my $document = qto/END/;
        you there
    END

    my $document = qq:heredoc/END/;
        and $b
    END

    my $result = qqx/ls -l bla/;
    my @bla = [+] @bar;
    my @list = @bla Z, @bar;
    say "yes" if $set1 (<) $set2;

    =for Pro-tip
    Pay careful attention to the following code

    if $a !< 4 {
        @bar».bla: "this &nice-func() thing";
    }
    my $bin = -0b01010;

    (1,2)[*/2];
    say $@foo.perl;
    m:s/foo/$bar/;
}

sub foo (Int:D $bla) is cached { }

Testing vim syntax files

Tue, 01 Sep 2009 13:13:00 +0000

Yesterday, Andy Lester opened an issue for vim-perl on github about adding an automated test suite. I’ve thought about doing something like this before, so last night got busy with prototyping a test harness.

What I’ve got so far (in my fork) is a test file that uses Text::VimColor to generate HTML and compare it against a reference HTML document to determine if the syntax file is doing its job. If a reference file can’t be found, it will create it and skip that test. Here’s what it looks like:

$ make test
prove -rv t
t/01_highlighting.t .. 
ok 1 - Correct output for t_source/perl/basic.t
ok 2 # skip Created t_source/perl/advanced.t.html
ok 3 - Correct output for t_source/perl6/basic.t
1..3
ok
All tests successful.

In case of failure, it will use Test::Differences to show you what’s wrong, and write the incorrect output to disk for you to inspect:

$ vim syntax/perl6.vim # make a bad change
$ make test
prove -rv t
t/01_highlighting.t .. 
ok 1 - Correct output for t_source/perl/basic.t
ok 2 - Correct output for t_source/perl/advanced.t
not ok 3 - Correct output for t_source/perl6/basic.t

#   Failed test 'Correct output for t_source/perl6/basic.t'
#   at t/01_highlighting.t line 77.
#
«output from Test::Differences showing the offending lines»
# You can inspect the incorrect output at t_source/perl6/basic.t_fail.html
1..3
# Looks like you failed 1 test of 3.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests

The only big downside to this is that 01_highlighting.t tests all the source files in one go. You currently can’t tell it to only test one specific file.

More docs added

Mon, 17 Aug 2009 23:52:00 +0000

I’ve added Jonathan Scott Duff’s introduction to Perl 6 regexes to Perl6-Doc as perlreintro. In addition I put in a first draft of a perlobjintro. Neither of these correspond to names from the set of Perl 5 man pages, which is fitting since they are not directly based on them. So, Perl6-Doc now has 4 man pages, the aforementioned two as well as perlintro and perlsyn. I’ll be carefully adding to and polishing these in the near future. I’m hesitant to start on man pages relating to modules, IO, laziness/iterators, and other very important features, as the specs (and implementation) for those are still shifting heavily from month to month.

I’ve been thinking about using the Perl Table Index as initial source material for the magical syntax recognition feature, the implementation of which has remained elusive for a while. The current implementation is just a proof-of- concept with only a few terms as of yet. However, Damian Conway just published a significant update to Synopsis 26 on Perl documentation, focused on tying docs to code, so it might be time to figure out some Pod-ish source format for the terms to be stored in. Writing Pod is a breeze, so it should be easy to contribute new terms.

grok also needs more love. I’m not as happy as I’d like to be with some of its implementation details, most notably how it picks out functions from Synopses 29 and 32, using simple regexes (to be fair, this is exactly what good ol’ perldoc -f does). It should parse the Pod and pick out parts of the document tree instead, I think. There are also still a few visual differences between textual output of Pod 5 and Pod 6 that I have to address. Pod 5 textual output is pleasingly indented (gradually depending on the current heading level), which makes it very easy to follow, while Pod 6 is not.

As a sidenote, I believe I’ve found what was causing intermittent FAIL reports from CPAN testers. Some machines had a pretty old version of Pod::Simple which didn’t take well to subclassing. Another failure was caused by an old Pod::Parser (deprecated, but relied on by Pod::Xhtml) version which didn’t recognize =encoding directives. The relevant distributions now depend on newer versions of those troublesome modules.

Tagging, FAIL, regexes

Wed, 29 Jul 2009 19:03:00 +0000

I’m currently working on making grok index all X<> (and maybe C<>) tags in Pod documents. The user will then be able to look them up and see which docs contain the search term in question.

Gábor Szabó has already done something like this for the Perl 6 Synopses. His implementation only looked for formatting codes using the standard <> format, but I just patched it to look for << >>, <<< >>>, and «» tags as well. Ideally I would like to use a Pod parser to get the tags, but it’s just using regexes for now. Gábor also asked me to have grok look up the predefined subrules from Synopsis 5, which I think would be a good addition.

For a while, I’ve been getting some rare FAIL reports about grok from CPAN testers. It seems to be two problems that always manifest themselves at the same time. One of them is obviously because the tester has an old version of Pod::Parser which doesn’t understand the =encoding directive, but the other one is more mysterious. I’ll have to do some more digging, as I haven’t been able to reproduce it on my machines.

Jonathan Scott Duff pointed me to his unpublished introduction to Perl 6 regexes which I could use as a starting point for a perlretut man page. I haven’t got it into a releasable state yet, but it will soon.

Grok refactor

Fri, 24 Jul 2009 13:30:00 +0000

As the title suggests, I reorganized the grok a bit, in addition to adding a few new features.

The code is now more modular, and much easier to maintain. I added support for looking up individual functions/methods from all the chapters of Synopsis 32. In other news, Perl6::Doc 0.42 is just out, which includes first drafts of perlintro and perlsyn documents, which grok will now find.

This brings the total number of things known to grok up by almost a hundred:

$ grok -i|wc -l
613

On to the man pages

Wed, 22 Jul 2009 18:31:00 +0000

This week I’ve been working on the perlintro document found in the pugs repository, as well as porting Perl 5’s perlsyn. These are the most “basic” man pages about the language, and should be ported first (especially since many of the more specific bits in Perl 6 are still in flux).

As for perlintro, I can see a few more-than-trivial things that need changing. For one thing, the introduction to regular expressions is very casual, but Perl 6’s regex/grammar support has seen a major overhaul. Simply translating the examples to the Perl 6 equivalent seems to just make them longer, which gives the wrong impression. Focusing less on quick-and-dirty examples and more on the big picture might help the newcomer here, as grammars are integral to Perl 6, not just some special strings with weird syntax as they were in ol’ Perl 5.

A lot of special cases and “warts” have been replaced with new, exciting concepts that show up again and again in Perl (laziness, autothreading, all blocks are closures, everything is an object) that need to be touched upon. You could say that in Perl 6, the harmony / idiosyncrasy ratio has been lowered, and that’s a very good thing. Some of these can be saved for perlsyn, but nonetheless, they need to be introduced well.

In other news, I noticed that perldoc.perl.org has seen a facelift. I wonder if the software behind it might be used to power a Perl 6 doc site sometime in the future.

Lucky 0.13

Thu, 16 Jul 2009 06:42:00 +0000

I just uploaded grok 0.13 to PAUSE. It has the things I mentioned in my last post, plus some bug fixes.

grok can look up quite a few things now (527 including documents such as Synopses), but many of the answers it provides lack thoroughness. According to my project schedule, it’s time to start writing new documentation. That means I can focus on making these answers better.

I should also write/update a tutorial pretty soon, though I’m not exactly sure where I’ll start. I could update the perlintro in the Pugs repository and add it to Perl6::Doc, or I could use the Perl 6 part of the book Perl 6 and Parrot Essentials which was donated to the Perl Foundation by O’Reilly. That one is a bit longer, so I might just use it for a more detailed introduction (something like Perl 5’s perlsyn).

P.S. Since I’ve got the Pod 5 ANSI-color renderer mostly working now, users of Perl 5’s perldoc who like colors might want to install Pod::Text::Ansi and put the following in their ~/.bashrc (or equivalent):

export PERLDOC="-MPod::Text::Ansi"

Half way there

Sat, 11 Jul 2009 01:54:00 +0000

With Summer of Code’s mid-term evaluations coming up, some interesting things are about to happen to grok. I was contacted by Herbert Breunung, author of Perl6::Doc (formerly Perl6::Bible), which is a project that shares some of grok’s goals.

He told me that he doesn’t have enough time to maintain his project anymore and would like some sort of merge to happen. I said I’d look into it. His project bundles all the Synopses, Apocalypses, Exegeses, and some other Perl 6 documentation, and ships with a perldoc wrapper for reading them. What I would like to do is to move all documentation out of the grok distribution and update/add more to Perl6::Doc while eliminating the perldoc wrapper (in favor of grok).

He also brought to my attention the Perl Table Index, a sort of Perl 6 glossary. Ahmad Zawawi has just patched his Padre::Plugin::Perl6 to feed this index to grok. I will probably port the Perl Table Index to the Perl6::Doc distribution and make grok look things up in it.

I’ve started writing a Pod::Text subclass as well as amending Perl6::Perldoc::To::Ansi to conform to a consistent color scheme when using the default ANSI-colored output.

Something which grok should eventually be able to do is recognize arbitrary Perl 6 syntax (at very fine level of granularity, that is) and tell you what it means. A first stab at this will be to simply include a table of some common ones and look those up. Stuff like my, +, and so on. Doing this reliably is the original inspiration for the u4x project (Userdocs for Christmas), of which grok is a part.

I hope to make a release of grok and Perl6::Doc shortly which will include all of the above.

grok 0.09 is out

Wed, 01 Jul 2009 19:33:00 +0000

The code is starting to take a more stable form. I’ve prepended an underscore to all private/internal subroutines and documented the rest. Perl authors wishing to use grok’s functionality will now have an easier time doing so.

I’ve also added various author tests to the distribution to help keep the code in shape (or at least remind me when it’s not), notably Test::Pod, Test::Pod::Coverage, and Test::Perl::Critic.

Since my last blog post, grok has gained a few features. It can print the name of the target file (like perldoc -l), print an index of known documentation files, output xhtml, and detect whether the target file has Pod 5 or Pod 6 in it. It’s also got some Win32 fixes and more informative error messages.

Pretty soon I will start bringing in some more docs to bundle with it. Tutorials and such. I will also most likely implement function documentation lookup (like perldoc -f) in grok.

grok update

Fri, 26 Jun 2009 05:54:00 +0000

Some things have kept me very busy lately and I’m a bit behind on my GSoC schedule. I’m starting to catch up now, though.

First of all, as of version 0.05, you can now easily install grok from the command line on most operating systems like so:

$ cpanp -i App::Grok

It can handle Pod 5 files now (via Pod::Text) as well (which most of the Perl 6 Synopses are written in). The ANSI-colored output looks a bit different than the Pod 6 output because Pod::Text is very conservative in its use of colors. I’ll probably make them more similar later to keep it consistent.

When called interactively, grok now uses your system’s pager to view the output by default.

It will now treat an argument as a Pod 6 file to read if said argument doesn’t match a known documentation target. As for the targets, the only known ones so far are the synopses (which are bundled with grok for now), so you can do things like:

$ grok s02
$ grok s32-rules

First GSoC post

Wed, 27 May 2009 17:25:00 +0000

I officially started on my Google Summer of Code project (project details here) last weekend.

I’ve been tasked with writing a perldoc equivalent for Perl 6. I’ve decided to write it in Perl 5 for now, since it’s already got Perl6::Perldoc, which is a fast and feature-complete parser for the Perl 6 version of Pod (see specification), as well as lots of other useful CPAN modules which I won’t have to rewrite in Perl 6 (yet).

The program will be called grok (repository). So far it’s just a barebones command-line reader for Pod 6, but I did create an ANSI-colored terminal renderer for Perl6::Perldoc. It might be too colorful for some people’s taste currently, so maybe I’ll tone it down a little. I think Ruby’s ri documentation reader is the only such tool which colors the output, and I thought it was a nice enough feature (one of many!) to copy. I’ve also been looking at pydoc and javadoc in search of interesting features to implement.

The plan is to make it more modern and extensive than perldoc. One interesting thing I might end up doing is making use of STD (the standard Perl 6 grammar) to parse arbitrary syntax, so you could do stuff like grok '[*]' and the program would tease the expression apart and show you documentation for both the [] and the * operators.

Some portion of the project also includes writing new documentation for Perl 6. I haven’t written any yet, but I bet it will be fun. Before doing so, we first have to figure out how to organize the the docs. Since Perl 6 is a specification, it’s not as obvious as with Perl 5, where the docs are included (sometimes inline) with the implementation.