tag:blogger.com,1999:blog-1023145445582315802024-03-13T03:03:13.174-07:00How does this thing work again?asdfhttp://www.blogger.com/profile/10583562613594805182noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-102314544558231580.post-28004030039423157482018-12-09T11:46:00.001-08:002022-04-03T13:37:24.870-07:00The crafting of a Figurine, or how to make a library in the new Dotnet SDK world<i>This post is part of the <a href="https://sergeytihon.com/2018/10/22/f-advent-calendar-in-english-2018/" target="_blank">F# Advent Calendar for 2018</a>. Thanks as always to <a href="https://twitter.com/sergey_tihon" target="_blank">Sergey Tihon</a> for organizing this yearly tradition :)</i><br />
<i></i><br />
<a name='more'></a><br />
<i><br /></i>
There's been a lot of motion in the .Net ecosystem overall the couple years, first with the announcement of .Net Core as a runtime and then with the rising popularity of the dotnet SDK and CLI as a way of interacting with your projects. While all the old knowledge about MSBuild and projects is still viable, I thought that I'd walk through some of the habits and practices I use to build libraries in this new work. To do this, I started making <a href="https://github.com/baronfel/Figurine" target="_blank">Figurine</a>.<br />
<br />
<i>A lot of this post is covered in a live stream that I did a week ago, so if you learn more visually you can go to the <a href="https://www.twitch.tv/videos/343758053##" target="_blank">Twitch stream</a> or the <a href="https://www.youtube.com/watch?v=Gug-hoOrFeg" target="_blank">YouTube version</a> to check it out.</i><br />
<i><br /></i>
<br />
<h2>
Motivation</h2>
<i><br /></i>
In the F# community we all know and love <a href="https://github.com/lefthandedgoat/canopy" target="_blank">Canopy</a>, the library that provides a nice DSL for orchestrating Selenium. In fact, my post two years ago about <a href="https://chethusk.blogspot.com/2016/12/a-host-of-issues-migrating-from.html" target="_blank">converting the F# UserVoice Suggestions to GitHub</a> used Canopy to power that whole migration. But these days we have a choice in browser automation tools, and the new kid on the block is <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>, a tool that uses WebSockets to drive instances of Chrome(ium) using the Debug Tools API. My thought was that it would be easy to create a library that had the same ease of use as Canopy but for this Puppeteer, and thus Figurine was born.<br />
<br />
<h2>
Setup</h2>
<div>
The first thing I knew I'd need is a good Continuous Integration pipeline. If Figurine was going to be a successful project, I needed not just the library itself but a consistent way to test, build, package, check, and publish the library, if for no other reason than to ensure that maintenance of the library wouldn't be a hassle to me. For this purpose I naturally chose to base the library off of <a href="https://twitter.com/Jimmy_Byrd" target="_blank">Jimmy Byrd's</a> <a href="https://github.com/TheAngryByrd/MiniScaffold" target="_blank">MiniScaffold</a> project. This project is a template for the <i>dotnet new </i>CLI command that provides a laundry list of features including<br />
<ul>
<li>a full-featured build/CI pipeline with <a href="https://fake.build/" target="_blank">FAKE 5</a></li>
<li>consistent, repeatable package dependency management with <a href="https://fsprojects.github.io/Paket/" target="_blank">Paket</a></li>
<li>automated versioning with RELEASE_NOTES.md</li>
<li>testing with Expecto</li>
<li>code coverage with AltCover</li>
<li><a href="https://github.com/dotnet/sourcelink/" target="_blank">Sourcelinking</a> of symbols/pdbs</li>
<li>Nuget packaging</li>
<li>GitHub releases</li>
<li>code formatting with Fantomas</li>
</ul>
<div>
all without setting anything up other than a name and a GitHub API token. The one major missing point is automated documentation generation and publishing, but there's even a work-in-progress pull request for that work already.</div>
<div>
<br /></div>
<div>
Getting started was a snap, I just had to ensure that I had the latest version of the template installed and then make my new project.</div>
<div>
<script src="https://gist.github.com/baronfel/5667a97c0d62fbf98a7787156ab677b1.js?file=init-project.sh"></script>
</div>
<div>
<br />
<h2>
Let's get hacking</h2>
</div>
</div>
<div>
Next I had to add the .Net version of the Puppeteer drive, <a href="https://github.com/kblok/puppeteer-sharp" target="_blank">Puppeteer Sharp</a>. This package is very active and exposes a <i>ton </i>of functionality, most of which I just needed to wrap in a way that evoked Canopy. Adding the package was very easy using Paket, I just ran 'mono .paket/paket.exe add PuppeteerSharp -i' and selected to install the package in my main Figurine project.</div>
<div>
<br /></div>
<div>
Once I had the dependency installed, I did a bit of reading in their API documentation. It seemed that there were three major concepts in Puppeteer:<br />
<ol>
<li>The 'Browser' is the overall connection to the browser instance that you're driving,</li>
<li>The 'Page' is your main entry point into interacting with a particular web page, and</li>
<li>The 'ElementHandle' is the main point for doing interaction with DOM elements, like reading properties, clicking, writing properties, etc.</li>
</ol>
<div>
From this, I thought of a few things immediately:<br />
<ul>
<li>I wanted tests to be stateless as much as possible, so no global configuration</li>
<li>I wanted test to be synchronous by default for easier reading (negotiable)</li>
<li>I wanted a user to not have to care about the internals of Pages/Browsers/etc.</li>
</ul>
<div>
So I immediately thought of a context/configuration type that would be a parameter to most functions and allow me to abstract the level of the API away from the page/element mechanics:<br />
<br /></div>
</div>
</div>
<div>
<script src="https://gist.github.com/baronfel/5667a97c0d62fbf98a7787156ab677b1.js?file=Types.fs"></script>
</div>
<div>
<br />
The idea was to give the user a way to create this semi-opaque BrowserCtx type from a set of LaunchOptions and then free them from handling the minutia of the browser/page duality. In the general case, right now, a user will spawn one BrowserCtx per test and operate on it for the duration of the test, like so:<br />
<br />
<br />
In the future I'd like to expand on this even further so that you could create a context and spawn multiple pages, one per test, from it more easily, because there's not really a need to create an entire browser. That's just for convenience right now.<br />
<br />
<h2>
Polishing it up</h2>
</div>
<div>
Moving on, I wrote some tests with the functions and parameters I'd need to test some simple functionality, like navigating to a URL, waiting for a particular condition, and finding elements by selectors. Naturally these all broke, but defining my use cases in terms of the end result was helpful in figuring out what kind of information I'd need in the BrowserCtx type. I then started filling in the implementations, using the API documentation for Puppeteer as a guide. It's all very rough now, but I ended up with the following basic functions for version 0.1.0:<br /><ul>
<li>click an element</li>
<li>get the current url</li>
<li>find all elements matching a selector</li>
<li>find a single element by a selector</li>
<li>run some arbitrary js</li>
<li>read the value from an element</li>
<li>navigate to a url</li>
<li>wait for a condition to be true on the page</li>
<li>write a value to an element</li>
</ul>
<div>
This was enough for me to do several useful tests, and is a good base to build on. Once I was happy with the API surface, I documented the functions briefly (here's an example from Fuget.org):</div>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-O6nIDwwSbLQ/XA1ukhTqTyI/AAAAAAAApUg/5H38NyrvbUsUzGav1yoKQTzk3omWiDWOACLcBGAs/s1600/api%2Bdoc.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="88" data-original-width="673" height="41" src="https://2.bp.blogspot.com/-O6nIDwwSbLQ/XA1ukhTqTyI/AAAAAAAApUg/5H38NyrvbUsUzGav1yoKQTzk3omWiDWOACLcBGAs/s320/api%2Bdoc.PNG" width="320" /></a></div>
<div>
It's a start, alright?</div>
<div>
<br /></div>
<div>
At this point all that was left to do in my little library was publish the package to nuget and coin a github release. Thanks to MiniScaffold, this only required three things:<br /><ol>
<li>Establishing an account on Nuget.org (I had already done this)</li>
<li>Making a Github Personal Access Token to create the release, and setting it to the GITHUB_TOKEN environment variable</li>
<li>running './build.sh Release'</li>
</ol>
<div>
At the end of that process I could go to <a href="https://www.nuget.org/packages/Figurine/" target="_blank">Figurine's Nuget page</a> and the <a href="https://github.com/baronfel/Figurine/releases/tag/0.1.0" target="_blank">GitHub releases page</a> to see the results of my work.</div>
</div>
<div>
<br /></div>
<h2>
Wrapping up</h2>
<div>
At the end of this process I got a lot for free from MiniScaffold and its integrations with other dotnet tools:</div>
<div>
<ul>
<li>Appveyor/Travis CI builds (which are breaking right now because I need an environment-agnostic way to download chrome, but....)</li>
<li>Release Management</li>
<li>Code Coverage (checkout the docs/ folder of the repo for the coverage report)</li>
</ul>
<div>
But there's still more that I need to do to really polish off Figurine, not all of which is actual code! I need to</div>
</div>
<div>
<ul>
<li>Write actual documentation and host it automatically</li>
<li>Work on the nuspec to add in better descriptions, icon, etc</li>
<li>Get the CI builds working consistently</li>
<li>Expand the capability of the library itself</li>
</ul>
<div>
But each of these things is doable, and more importantly I got a head-start on all of them _for free_ by using the work of other members of the community. And for the first point, I can work with Jimmy and other maintainers to get documentation-generation into the MiniScaffold template itself and then everyone will get the benefit of docs out-of-the-box.</div>
</div>
<div>
<br /></div>
<div>
I'm very glad that it's so easy to get the grunt work of productionalizing a library out of the way these days. Between dotnet sdk templates, the mainstreaming of sourcelink, and the unification onto .netstandard2.0, it's never been less hassle to share your code with others, so get out there and share some of your own!</div>
<div>
<script src="https://gist.github.com/baronfel/5667a97c0d62fbf98a7787156ab677b1.js?file=navTest.fs"></script>
</div>
asdfhttp://www.blogger.com/profile/10583562613594805182noreply@blogger.comtag:blogger.com,1999:blog-102314544558231580.post-37158708586371159932016-12-01T22:00:00.001-08:002022-04-03T13:37:33.314-07:00A Host of Issues: Migrating from UserVoice To GitHub Issues with Canopy and OctoKit.net<i>This post is part of the <a href="https://sergeytihon.wordpress.com/2016/10/23/f-advent-calendar-in-english-2016/" target="_blank">F# Advent Calendar for 2016</a>. Thanks to Sergey for putting this on, it's always wonderful to see what everyone highlights this time of year!</i><br />
<div>
<i><br /></i></div>
<div>
<i></i><br />
<a name='more'></a></div>
<h3>
</h3>
<div>
On October 6th, <span class="fn" style="border: 0px none; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><a href="https://twitter.com/k_cieslak" target="_blank">Krzysztof Cieslak</a></span> had had enough. He'd grown tired of spending time working on submissions to UserVoice for F# language enhancements and after some prodding submitted <a href="https://fslang.uservoice.com/forums/245727-f-language/suggestions/16529041-close-user-voice-in-favor-of-github-issues" target="_blank">THE SUGGESTION THAT WOULD LIVE IN INFAMY</a>. It immediately generated some frenzy, and I'd had a bad day at work, so on a whim I decided to finally learn <a href="https://lefthandedgoat.github.io/canopy/" target="_blank">Canopy</a>, a Selenium library for F#, and try my hand at a first pass of a migration of issues from UserVoice to GiHub Issues.</div>
<div>
<span style="color: black;"><br /></span></div>
<div>
Canopy is really a fantastic library. In essence it's a tightly-crafted DSL on top of the Selenium WebDriver library, which allows for easier automation of web browsers though code. It's most commonly used for UI acceptance tests in my experience, but we're going to pervert its intentions a bit.</div>
<div>
<br /></div>
<div>
But first, some links:</div>
<div>
<ul>
<li>The old uservoice is here: <a href="https://fslang.uservoice.com/forums/245727-f-language">https://fslang.uservoice.com/forums/245727-f-language</a></li>
<li>The new language design repo is here: <a href="https://github.com/fsharp/fslang-suggestions">https://github.com/fsharp/fslang-suggestions</a></li>
<li>The repo for the code that ended up generating the new language design repo is here: <a href="https://github.com/dsyme/fsharp-lang-suggestions-scripts">https://github.com/dsyme/fsharp-lang-suggestions-scripts</a></li>
</ul>
<div>
<br /></div>
</div>
<div>
And please don't judge to code quality too harshly, <a href="https://twitter.com/cloudroutine" target="_blank">Jared</a> and I were kinda blazing through this thing in off hours!</div>
<div>
<br /></div>
So really what we had to accomplish was three things:<br />
<div>
<ol>
<li>Discover the list of all issues on UserVoice</li>
<li>Pull over metadata for each of the issues:</li>
<ol>
<li>Submitter</li>
<li>Content</li>
<li>Date</li>
<li>Votes</li>
<li>Comments</li>
<ol>
<li>Submitter</li>
<li>Date</li>
</ol>
<li>Official Responses</li>
<ol>
<li>Responder</li>
<li>Date</li>
<li>Content</li>
</ol>
</ol>
<li>Use that metadata to create a matching issue on GitHub</li>
</ol>
<div>
<br />
So to start with, after hashing out some requirements we landed with the following set of base models:</div>
</div>
<div>
<script src="https://gist.github.com/baronfel/d253e43c9299ad1055806a37d8b8ce39.js?file=model.fs"></script></div>
<div>
<br /></div>
<div>
With these models in hand, we can now go parse sections of the each page out to get each Idea. All of the code directly related t scraping the pages is <a href="https://github.com/dsyme/fsharp-lang-suggestions-scripts/blob/master/src/Scraping.fs" target="_blank">here</a>. An interesting point is that there's no master list of all issues for a forum, you've got to go scrape through each different category of issues to get the entire list, so that's what we've done in the discoverIdeas function.</div>
<div>
<script src="https://gist.github.com/baronfel/d253e43c9299ad1055806a37d8b8ce39.js?file=links.fs"></script></div>
<div>
<br />
Next was the grunt work of cycling through each of the links we just found and parsing them. This is a hairy chunk, but relatively straightforward:<br />
<br /></div>
<div>
<script src="https://gist.github.com/baronfel/d253e43c9299ad1055806a37d8b8ce39.js?file=ideas.fs"></script></div>
<br />
After everything was parsed, we did a quick Json.Net dump of the Idea list into a file so we wouldn't have to scrape that again. It took about 20 minutes to scrape all of UserVoice because Canopy seems to implicitly have a single webdriver context. I thought about parallelizing by looking into passing contexts around to use, but I couldn't find a way to do that and once we had the data everything else fell into place.<br />
<br />
Having secured the data, we then needed to make markdown issue templates to render the ideas into a form that would look good on the GitHub Issue. My preferred templating library in .Net is <a href="http://dotliquidmarkup.org/" target="_blank">DotLiquid</a> for its simple setup and reliability. If you'd like to see what the template looks like, take a quick detour and check it out <a href="https://github.com/dsyme/fsharp-lang-suggestions-scripts/blob/master/templates/_idea_submission.liquid" target="_blank">here</a>. Simple, no? I did have to do a bit of configuration to get the templating system to recognize F# records, but luckily that work had already been done by (I think) Tomas Petricek and Henrik Feldt. Basically what we have to do to use the templates in a nice way is register the public members of any type we want to use as a model in the template, and there are some special cases around Seq, List, and Option that have to be handled for dotLiquid to work. The code looks like this:<br />
<br />
<div>
<script src="https://gist.github.com/baronfel/d253e43c9299ad1055806a37d8b8ce39.js?file=template.fs"></script></div>
Finally came the hard part - interacting with the GitHub api. Thankfully, we had OctoKit.net available to smooth over calling the api directly, but as everyone knows the GitHub api is a bit aggressive when it comes to rate limiting consumers. You're limited to 20 calls per minute, and a daily allotment of 5000 calls overall. The initial version of the upload code was incredibly parallel, queuing up all the work making use of Asyncs all over the place. That code bombed almost immediately due to burning through our api allowance. We eventually settled on including delays periodically to slow down the overall rate of upload, but the rate limit made the upload process crawl to hours instead of what we had hoped would be minutes. At this moment there are two separate attempts to manage throttling more intelligently in the Github.fs file, however none were good enough. In fact, months after we did this work, I've started a MailboxProcessor-based version that would handle caps on my fork of the repo but it's still not up to par. So if anyone has any pointers to intelligently manage rate limiting in a general/intelligent way that changes based on output of the api client after each call, let me know!<br />
<br />
The code for uploading to GitHub isn't really complicated at all, and lives <a href="https://github.com/dsyme/fsharp-lang-suggestions-scripts/blob/master/src/Github.fs" target="_blank">here</a>. It's mostly a straightforward mapping of Idea -> Issue GitHub Model and then we POST that model up. We do do a few interesting things like assign labels and close the issue automatically if the issue was closed/rejected/etc on UserVoice, as one would expect.<br />
<br />
Overall, I'd say that two people spent ~10 hours each working on this, and A LOT of that time was waiting on uploads to break. We spent some time tweaking the issue templates for formatting, but that was a very iterative process. Canopy made it incredibly simple to grab the data we needed to do the migration, DotLiquid made it easy to render nice markdown for the Issues, and OctoKit...OctoKit wasn't the worst.<br />
<br />
My overall hope with this post is to show that F# is as great for relatively simple, one-off tasks as it is for more in-depth projects like dependency management, finance, microservices, and any number of other realms (though I do use it there as well). So I encourage everyone who's looking for ways to include it at work to try and tackle your next annoying problem with a fsx script. That's how this project started, until it grew too unwieldy.<br />
<br />
Happy Holidays, and Merry F#ing!<br />
<br />asdfhttp://www.blogger.com/profile/10583562613594805182noreply@blogger.comtag:blogger.com,1999:blog-102314544558231580.post-78887524622596895362016-03-28T12:56:00.001-07:002022-04-03T13:37:42.512-07:00Adventures in AWS: Nested IIS configurationsI was automating some deploys of IIS apps in AWS ElasticBeanstalk using TeamCity, and had passed some Teamcity environment variables into the AWS EB environment configuration only to find that the variables did NOT exist in my app's local Web.config post-deployment!
I searched around in IIS for a bit and found that they were actually being set at the web.config for the hosting web site, which by convention is 'Default Web Site'. I spent a little while wracking my brain for how to get access to those values in a way that wasn't file-read-based, because the executing user account doesn't have access outside it's hosting directory, when I came across a reference to the <br />
System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration<br />
static method. When used with the name of a web site, this will read and parse the config into a <br />
System.Configuration.Configuration<br />
object. So at last, I could use the following code to merge the two sources of settings:
<script src="https://gist.github.com/baronfel/f604b24b430c842da538.js?file=Config_example.fs"></script>asdfhttp://www.blogger.com/profile/10583562613594805182noreply@blogger.comtag:blogger.com,1999:blog-102314544558231580.post-39969151093992385032014-07-14T20:04:00.004-07:002022-04-03T13:37:54.178-07:00Parsing RAML with FParsec, Part 2: The Spec That Launched A Thousand LOC<h3>
RAML, the Spec</h3>
<div>
So, what exactly does RAML look like? You can find the official spec on <a href="https://github.com/raml-org/raml-spec/blob/master/raml-0.8.md" target="_blank">GitHub</a>, but that seems a bit remote (Ha! Sorry, couldn't resist the DVCS pun :)). Let's take a look at what some actual specs would look like and see if we can build an intuition from that.</div>
<div>
<br /></div>
<div>
Here's an embed of a file that's in my repo that I'm using for testing:</div>
<div>
<br /></div>
<script src="https://gist.github.com/baronfel/8b1fbfae235e5084b17a.js?file=0.8_spec"></script>
<div>
<br /></div>
<div>
First off, RAML is an extension of YAML, which is a significant-whitespace, human-readable data interchange format. It also supports remote references, which <strike>may</strike> will likely be a pain to parse later. This isn't an exhaustive example of all the things RAML must support, but it's enough to get going.</div>
<div>
<br /></div>
<h3>
Baby Steps</h3>
<div>
My general approach with this project is going to be to take each item in line in this example, taking care to adhere to the spec as best as I can interpret for each type of element. I'm going to try to be pretty exhaustive, because I'm pretty sure this blog is going to be my own reference for what I was thinking when I'm crawling through git blame cursing at myself.</div>
<div>
<br /></div>
<div>
So, let's start out with the very first tag, the RAML spec number. It looks like this:</div>
<div>
<br /></div>
<script src="https://gist.github.com/baronfel/8b1fbfae235e5084b17a.js?file=ramlversion"></script>
<div>
<br /></div>
<div>
This looks very similar to a YAML version number; namely it's </div>
<div>
<ol>
<li>a special comment marker (#%), </li>
<li>a file type (RAML), </li>
<li>some whitespace, </li>
<li>and the version number. </li>
</ol>
Because we have these discrete elements, why not parse them out in the same way I just described? The nice thing about FParsec, and other libraries known as "applicative combinators", is that it allows us to do just that. We can define parsers for each of the elements above, chain them in sequence, and get some greater meaning out of the disparate parts.</div>
<div>
<br /></div>
<h3>
AST, Oh Me!</h3>
<div>
Before we jump into the parsing, we need to have some idea of what the final representation of our API will look like. This schema is an Abstract Syntax Tree, or AST. We'll define the shape of RAML in an AST, and then define a parser for a single part of the AST, joining them together one by one until we have a final, full implementation of the spec.</div>
<div>
<br /></div>
<div>
So! Starting from the very beginning of our RAML sample, the spec number, our AST would look like this in F#:</div>
<div>
<br /></div>
<script src="https://gist.github.com/baronfel/8b1fbfae235e5084b17a.js?file=ast"></script>
<div>
<br /></div>
<div>
Our spec, initially, will only be able to parse the version. So let's go to that.</div>
<div>
<br /></div>
<h3>
Our First Parser</h3>
<div>
FParsec comes with a multitude of parsers. Let's take a look at the first one we'll be using, skipString. Its signature is:</div>
<div style="text-align: center;">
<i>val skipString: string -> Parser<unit, 'u></i></div>
<div style="text-align: left;">
Let's break this down a bit. </div>
<div style="text-align: left;">
<ol>
<li>val skipString- this is just the name of the function . This parser is a public member of its containing module.</li>
<li>string -> - this parser is a function that takes a string and returns something else</li>
<li>Parser<unit, 'u> - this parser yields unit (or void), and some generic User State. This User State is something that we don't care about right now, so pretend it's of type unit as well, or void.</li>
</ol>
So how do we use this to do something useful? Well, if you combine the skipString parser with a string to expect, what it'll do is walk through the input stream and consume that string, and then throw it away, advancing us along. This is perfect for situations like we have here, where there's some useless string standing between us and the version number we want to parse. Let's try using the skipString parser generator function to generate a parser for the sentinel string for the version number.</div>
<div>
<br /></div>
<script src="https://gist.github.com/baronfel/8b1fbfae235e5084b17a.js?file=versionString_1"></script>
<div style="text-align: left;">
<b><br /></b></div>
<div style="text-align: left;">
Ok, we're set. Now, the whole point of FParsec is to allow us to chain building block like these together to make more complex constructs. We chain these together with a bunch of different operators. The ones I use most commonly, so far at least, are:</div>
<div style="text-align: left;">
<ul>
<li>.>></li>
<ul>
<li>This operator is used like this: parserA .>> parserB, and means "evaluate parserA, then evaluate parserB, and return the result of parserA"</li>
</ul>
<li>>>.</li>
<ul>
<li>This operator is used the same way, but returns the result of parserB instead.</li>
</ul>
</ul>
<div>
So, to make a full parser for the version string line in a spec, we'd want to chain together a parser for the sentinel (which we've done), a parser for the space, a parser for the version number, and a parser that advances us to the next line. Using the operators we just saw, that looks like this:</div>
<div>
<br /></div>
<script src="https://gist.github.com/baronfel/8b1fbfae235e5084b17a.js?file=versionString_2"></script>
<div style="text-align: left;">
If you read this in order, it reads:</div>
<div style="text-align: left;">
<ol>
<li>Evaluate "skipString "#%RAML"" and "ws", returning the former,</li>
<li>Evaluate that previous result and "pfloat", returning the latter,</li>
<li>Evaluate that previous result and "advanceToEOL" (which is really just an alias for blowing through the rest of the line) and return the former.</li>
</ol>
<div>
Which, in essence, is what we were trying to do with our stated goal of "Parsing a RAML version string". Now that we have this value, we have to get it into an instance of RamlDef, which FParsec does using the operator |>>, which means "apply the result of the Parser on the left (remember the result of a Parser<'a,'b> is 'a) as the first parameter to the function on the right. So if we have a function 'makeRamlDef' that accepts a float, we could do </div>
<div style="text-align: left;">
<br /></div>
<script src="https://gist.github.com/baronfel/8b1fbfae235e5084b17a.js?file=versionString_final"></script>
<div style="text-align: left;">
Which, as written, is read "the raml parser is the result of the version parser applied to the function makeRamlDef"</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
And now, if you go to an f# interactive, or a test project or something, and call the FParsec run method</div>
<div style="text-align: left;">
<br /></div>
<script src="https://gist.github.com/baronfel/8b1fbfae235e5084b17a.js?file=run"></script>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
You'll end up with "success"!</div>
<div style="text-align: left;">
<br /></div>
<h3 style="text-align: left;">
Conclusion</h3>
<div>
So now we've at least got a single parser. Next time we'll look at combining separate parsers for RAML spec data in sequence as we fill out more of the spec.</div>
<div>
<br /></div>
</div>
</div>asdfhttp://www.blogger.com/profile/10583562613594805182noreply@blogger.comtag:blogger.com,1999:blog-102314544558231580.post-7338802351066620082014-07-14T17:11:00.001-07:002022-04-03T13:38:00.497-07:00Parsing RAML with FParsec, Part 1: Background and Goals<h2>
Intro</h2>
<div>
Hi everyone, and welcome to what I hope will be an exciting journey into the world of parsers and combinators. </div>
<div>
<br /></div>
<div>
Yeah, those words terrify me too.</div>
<div>
<br /></div>
<div>
I had this crazy idea one day at work that it would be fantastic if I had some better tooling around our internal representation of a particular REST service. In general, these things are planned out ahead of time and contracts are negotiated, but even in the best of times some details can be left out and consistency in you API can be something that's hard to quantify.</div>
<div>
<br /></div>
<div>
So I went and did an initial search to see what the tooling was like and stumbled across <a href="http://raml.org/" target="_blank">RAML.org</a>, the landing page of the RESTful API Modeling Language. This seemed like a wonderful tool, with a spec that seemed well-geared towards encouraging API consistency by use of common <a href="https://github.com/raml-org/raml-spec/blob/master/raml-0.8.md#resource-types-and-traits" target="_blank">traits</a> and <a href="https://github.com/raml-org/raml-spec/blob/master/raml-0.8.md#security" target="_blank">security schemes</a>. It also has <a href="http://api-portal.anypoint.mulesoft.com/raml/api-designer" target="_blank">visualizers </a>and tooling around code generation, which can help reduce the time it takes to crank out the internals of your API. Anyway, check out RAML or any of its competitors if you're designing an API.</div>
<div>
<br /></div>
<div>
Now, work is mostly .Net languages, and most(all?) of the existing RAML tools are in more...dynamic languages. I thought it would be really helpful to have some sort of automated converter for .Net that would allow you to populate a structure with your API setup and generate a RAML-compliant description, at which point you could plug that into existing tooling, make changes, and design new APIs. You could also use this model to do code generation, analysis, etc.</div>
<div>
<br /></div>
<div>
So at this point the delusions of grandeur started to flow, and I thought "If I have a parser, I could totally generate a F# Type Provider using the RAML as the data source to allow for easy exploration and consumption of that API."</div>
<div>
<br /></div>
<div>
And then "If I have an explorable, strongly-typed representation of a REST endpoint, there are F# libraries for generating comprehensive test cases for various inputs, so I MIGHT even be able to exhaustively test a subset of RAML-compliant REST services AUTOMATICALLY!"</div>
<div>
<br /></div>
<div>
But all of these cases require the parser, and so I began to look at what tools I had available. F#, and functional languages in general, have developed a reputation for being good at parsing text into data. In addition, I'd developed a fancy for the language after working with the magnificent, cross-platform build tool <a href="http://fsharp.github.io/FAKE/" target="_blank">FAKE</a>. So it was that I stumbled upon <a href="http://www.quanttec.com/fparsec/" target="_blank">FParsec</a>, and so begins this series of blog posts.</div>
<div>
<br /></div>
<div>
So, to summarize, I hope to use FParsec and the RAML spec to:</div>
<div>
<ol>
<li>Generate a .Net native parser for the RAML spec that outputs a .Net type representing a REST endpoint and its constraints, methods, etc</li>
<li>Generate a Type Provider that you can direct at a RAML spec, either a URI or a local file, and use to explore that API at design time, and (stretch goal)</li>
<li>Develop a tool that, given an endpoint with sufficient constraints around the inputs and outputs of its routes, can be tested in some degree automatically for compliance to a matching RAML spec.</li>
</ol>
<div>
The next post should follow shortly, and we'll start with the basics of parsing using FParsec. Do note that I'm still quite new to this, so please join in with any comments or suggestions!</div>
</div>
asdfhttp://www.blogger.com/profile/10583562613594805182noreply@blogger.com