What I’ve got so far (in my fork) is a test file that uses Text::VimColor to generate HTML and compare it against a reference HTML document to determine if the syntax file is doing its job. If a reference file can’t be found, it will create it and skip that test. Here’s what it looks like:

$ make test
prove -rv t
t/01_highlighting.t ..
ok 1 - Correct output for t_source/perl/basic.t
ok 2 # skip Created t_source/perl/advanced.t.html
ok 3 - Correct output for t_source/perl6/basic.t
1..3
ok
All tests successful.

In case of failure, it will use Test::Differences to show you what’s wrong, and write the incorrect output to disk for you to inspect:

$ vim syntax/perl6.vim # make a bad change
$ make test
prove -rv t
t/01_highlighting.t ..
ok 1 - Correct output for t_source/perl/basic.t
ok 2 - Correct output for t_source/perl/advanced.t
not ok 3 - Correct output for t_source/perl6/basic.t

#   Failed test 'Correct output for t_source/perl6/basic.t'
#   at t/01_highlighting.t line 77.
#
«output from Test::Differences showing the offending lines»
# You can inspect the incorrect output at t_source/perl6/basic.t_fail.html
1..3
# Looks like you failed 1 test of 3.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests

The only big downside to this is that 01_highlighting.t tests all the source files in one go. You currently can’t tell it to only test one specific file.

I’ve been thinking about using the Perl Table Index as initial source material for the magical syntax recognition feature, the implementation of which has remained elusive for a while. The current implementation is just a proof-of-concept with only a few terms as of yet. However, Damian Conway just published a significant update to Synopsis 26 on Perl documentation, focused on tying docs to code, so it might be time to figure out some Pod-ish source format for the terms to be stored in. Writing Pod is a breeze, so it should be easy to contribute new terms.

grok also needs more love. I’m not as happy as I’d like to be with some of its implementation details, most notably how it picks out functions from Synopses 29 and 32, using simple regexes (to be fair, this is exactly what good ol’ perldoc -f does). It should parse the Pod and pick out parts of the document tree instead, I think. There are also still a few visual differences between textual output of Pod 5 and Pod 6 that I have to address. Pod 5 textual output is pleasingly indented (gradually depending on the current heading level), which makes it very easy to follow, while Pod 6 is not.

As a sidenote, I believe I’ve found what was causing intermittent FAIL reports from CPAN testers. Some machines had a pretty old version of Pod::Simple which didn’t take well to subclassing. Another failure was caused by an old Pod::Parser (deprecated, but relied on by Pod::Xhtml) version which didn’t recognize =encoding directives. The relevant distributions now depend on newer versions of those troublesome modules.

For a while, I’ve been getting some rare FAIL reports about grok from CPAN testers. It seems to be two problems that always manifest themselves at the same time. One of them is obviously because the tester has an old version of Pod::Parser which doesn’t understand the =encoding directive, but the other one is more mysterious. I’ll have to do some more digging, as I haven’t been able to reproduce it on my machines.

Jonathan Scott Duff pointed me to his unpublished introduction to Perl 6 regexes which I could use as a starting point for a perlretut man page. I haven’t got it into a releasable state yet, but it will soon.

This brings the total number of things known to grok up by almost a hundred:

$ grok -i|wc -l
613

A lot of special cases and “warts” have been replaced with new, exciting concepts that show up again and again in Perl (laziness, autothreading, all blocks are closures, everything is an object) that need to be touched upon. You could say that in Perl 6, the harmony / idiosyncrasy ratio has been lowered, and that’s a very good thing. Some of these can be saved for perlsyn, but nonetheless, they need to be introduced well.

In other news, I noticed that perldoc.perl.org has seen a facelift. I wonder if the software behind it might be used to power a Perl 6 doc site sometime in the future.

I should also write/update a tutorial pretty soon, though I’m not exactly sure where I’ll start. I could update the perlintro in the Pugs repository and add it to Perl6::Doc, or I could use the Perl 6 part of the book Perl 6 and Parrot Essentials which was donated to the Perl Foundation by O’Reilly. That one is a bit longer, so I might just use it for a more detailed introduction (something like Perl 5’s perlsyn).

P.S. Since I’ve got the Pod 5 ANSI-color renderer mostly working now, users of Perl 5’s perldoc(1) who like colors might want to install Pod::Text::Ansi and put the following in their ~/.bashrc (or equivalent):

export PERLDOC="-MPod::Text::Ansi"

He told me that he doesn’t have enough time to maintain his project anymore and would like some sort of merge to happen. I said I’d look into it. His project bundles all the Synopses, Apocalypses, Exegeses, and some other Perl 6 documentation, and ships with a perldoc wrapper for reading them. What I would like to do is to move all documentation out of the grok distribution and update/add more to Perl6::Doc while eliminating the perldoc wrapper (in favor of grok).

He also brought to my attention the Perl Table Index, a sort of Perl 6 glossary. Ahmad Zawawi has just patched his Padre::Plugin::Perl6 to feed this index to grok. I will probably port the Perl Table Index to the Perl6::Doc distribution and make grok look things up in it.

I’ve started writing a Pod::Text subclass as well as amending Perl6::Perldoc::To::Ansi to conform to a consistent color scheme when using the default ANSI-colored output.

Something which grok should eventually be able to do is recognize arbitrary Perl 6 syntax (at very fine level of granularity, that is) and tell you what it means. A first stab at this will be to simply include a table of some common ones and look those up. Stuff like ‘my‘, ‘+‘, and so on. Doing this reliably is the original inspiration for the u4x project (Userdocs for Christmas), of which grok is a part.

I hope to make a release of grok and Perl6::Doc shortly which will include all of the above.

Since my last blog post, grok has gained a few features. It can print the name of the target file (like perldoc -l), print an index of known documentation files, output xhtml, and detect whether the target file has Pod 5 or Pod 6 in it. It’s also got some Win32 fixes and more informative error messages.

Pretty soon I will start bringing in some more docs to bundle with it. Tutorials and such. I will also most likely implement function documentation lookup (like perldoc -f) in grok.

$ cpanp -i App::Grok.

It can handle Pod 5 files now (via Pod::Text) as well (which most of the Perl 6 Synopses are written in). The ANSI-colored output looks a bit different than the Pod 6 output because Pod::Text is very conservative in its use of colors. I’ll probably make them more similar later to keep it consistent.

When called interactively, grok now uses your system’s pager to view the output by default.

It will now treat an argument as a Pod 6 file to read if said argument doesn’t match a known documentation target. As for the targets, the only known ones so far are the synopses (which are bundled with grok for now), so you can do things like:

$ grok s02
$ grok s32-rules

I’ve been tasked with writing a perldoc equivalent for Perl 6. I’ve decided to write it in Perl 5 for now, since it’s already got Perl6::Perldoc, which is a fast and feature-complete parser for the Perl 6 version of Pod (see specification), as well as lots of other useful CPAN modules which I won’t have to rewrite in Perl 6 (yet).

The program will be called grok (repository). So far it’s just a barebones command-line reader for Pod 6, but I did create an ANSI-colored terminal renderer for Perl6::Perldoc. It might be too colorful for some people’s taste currently, so maybe I’ll tone it down a little. I think Ruby’s ri documentation reader is the only such tool which colors the output, and I thought it was a nice enough feature (one of many!) to copy. I’ve also been looking at pydoc and javadoc in search of interesting features to implement.

The plan is to make it more modern and extensive than perldoc. One interesting thing I might end up doing is making use of STD (the standard Perl 6 grammar) to parse arbitrary syntax, so you could do stuff like grok '[*]' and the program would tease the expression apart and show you documentation for both the [] and the * operators.

Some portion of the project also includes writing new documentation for Perl 6. I haven’t written any yet, but I bet it will be fun. Before doing so, we first have to figure out how to organize the the docs. Since Perl 6 is a specification, it’s not as obvious as with Perl 5, where the docs are included (sometimes inline) with the implementation.