Another Security Model

I had an interesting question when sitting with a client today. The Event Store supports internally role based security through ACLs they would prefer to use a claim based system with it. An interesting idea, is there a reasonably easy way to do this? Well yes but it requires a bit of coding around (would be a nice thing to have in a library somewhere *wink wink*)

The general idea with claims based security is that something else will do the authentication and the application will act only on a series of claims that are given. In this particular example they want to control access to streams based upon claims about the user and to do it in a reasonably generic way.

As an example for a user you may receive the following claims.

{
    organization : 37,
    department : 50,
    team : 3,
    user : 12
}

What they want is to be able to use these in conjunction with streams to determine whether or not a given user should have access to the stream (and to be reasonably dynamic with it.)

Obviously we will not be able to easily do this with the internal security (well you could but it would be very ugly) but it can be built relatively easily on top. It is quite common to for instance run Event Store only on localhost and to only expose a proxy publicly (this kind of thing can be done in the proxy, while not an ideal solution it can get us pretty close to what we want.)

If we just want to work with say a single claim “can read event streams” we could simply do this in the proxy directly and check the claim before routing the request. Chances are however you want to do quite a bit more with this and make it more dynamic however which is where the conversation went in particular what about per stream and per stream-type setup dynamically? Well we could start using the stream metadata for this.

For a resource (stream metadata)

{
    organization : 37,
    department : 50,
    team : 13,
    user : 121
}

Now we could try taking the intersection of this with the claims provided on the user.

The intersection would result in

{
    organization : 37,
    department : 50,
}

We might include something along the lines of

{
    approve = "organization,department",
    somethingelse = "organization,department,team"
    delete = "user"
}

Where the code would then compare the intersection to the verb you were trying to do (must have all). This is a reasonably generic way of handling things but we can one step further and add in a bit more.

/streams/account-defaultsetting

{
    approve = "organization,department",
    somethingelse = "organization,department,team"
    delete = "user"
}

This is now defined in a default which will be merged with the stream metadata (much like how ACLs work). If a value is provided in the stream metadata it will override the default (based on type of stream). This allows us to easily setup defaults for streams as well. The logic in the proxy is roughly as follows:

Read {type}-defaultsetting (likely cached)
Read streammetadata
Merge streammetadata + {type-defaultsetting} to effective metadata
Calculate intersection of effective metadata with user info
Check intersection vs required permission for operation

This provides a reasonably generic solution that is quite useful in many circumstances. The one issue with it is that if someone roots your box they can access the data directly without permissions as they can bypass the proxy and talk directly on localhost (to be fair your probably have bigger problems at this point). It is however a reasonable solution for many situations.

Ouro’s Birthday

We are now over 3 years working on Event Store! Ouro’s second birthday will be happening in London Sept 17. In general Ouro’s birthday is always a lot of fun!

This year we will have some short talks by varying contributors on the OSS side (what they have done, what they are working on etc) and I will do a talk on major functionality completed/on the way including showing off some new goodies.

There will be plenty of people from the community there and I am sure lots of good discussions. After there will be a bit of a celebration with free food/beer and of course some ouro swag. Come on out and hang out for the evening! RSVP required: http://geteventstore.com/two-years-on/#tickets

Sublime is Sublime Closing

Well its an early morning. I can blame the travel from London for that. I managed to struggle through to the end of the second period watching the canadiens game last night. I was a bit worried entering the third but was quite happy to see they won when I woke up :)

In this post I just want to sum up the other posts from the sublime series as well as add a few tidbits. In the post series we have learned how to setup sublime for .net development. We have covered how to setup project/solution support. How to get intellisense and some basic refactoring. Even how to get automated builds and tests running (all in linux).

We have also looked at a lot of other things that are built on top of sublime that are fairly useful if you are doing other types of development such as javascript or html5. Many of these tools far outclass the Visual Studio equivalents and are usable with many other environments (such as using a ruby backend).

I have personally given up on using Visual Studio as a whole. I will however keep a vm with it on it for some very specific tasks that it does well (such as line by line debugging). These are not things I use in my daily worksflow but are nice to have when you absolutely need them.

Some other changes have come about in the use of sublime as my primary editor. A big one is that when I am writing one off code (which I do alot) I do not bother creating project or solution files any more. I instead just create C# files then either run the command line directly to the compiler or create a small make file. It sounds odd but its actually much simpler than creating project/solution files overall.

There will also be much going on in this space coming up. As of now the sublime plugin supports maybe 20% of what omnisharp is capable of. There will be quite a bit of further support coming in. As an example I was looking the other day at supporting run tests in context from inside of sublime (in test->run test, on fixture->run fixture). There is also much coming in for refactoring support and my guess is that you will see even more coming in on this due to nrefactory moving to roslyn. I think within a year you will find most of this tooling built in.

Another thing that I added to sublime though there isn’t really an official plugin for it yet is sublime repl + script cs. I find it quite common to grab a function and work it out in the repl first and then move it back into the code. A perfect example of this happened to me while in London. I was trying to combine to Uris and was getting some odd behaviour. Three minutes in the repl showed exactly what the issue was.

Moving to sublime will change the way that you work a bit but is definitely worth trying. Remember that a primary benefit of working in this way is that everything that you are doing is a composition of pieces that will also apply to any other code you happen to be working on (whether its C/Ruby/Erlang/even F#).

Banking Example Again

I was reading through this yesterday on my way out of London. Go on take a minute and read it.

http://hackingdistributed.com/2014/04/06/another-one-bites-the-dust-flexcoin/

I do find it funny that the bitcoin exchanges were taken down by such things but the article is pretty ridiculous in how it presents its problem/solution. Banks don’t actually work as described in this post. There is not a column “balance” in an account table as presented unless the developers just had no clue what they were doing.

mybalance = database.read("account-number") newbalance = mybalance - amount database.write("account-number", newbalance) dispense_cash(amount) // or send bitcoins to customer - See more at: http://hackingdistributed.com/2014/04/06/another-one-bites-the-dust-flexcoin/#sthash.esawajq9.dpuf

This is absurd your balance while perhaps being denormalized on your account is really the result of an equation (summation of the value of your transactions). All of these problems discussed would just go away if the system had been designed to record a journal properly (and as the journal is append only most other issues would go away).

I have always hated that the typical example of distributed transactions is transfering between two accounts. Banks don’t work this way!

Sublime is Sublime 12

OK so we are at the last blog post in the sublime series. I have I hope saved the best one for last. One of the largest questions I have received during this series is how do I get intelisense and that R# is awesome because it supports things like goto definition and rename…

In this post we will add all of these features to sublime. There is a great project out there called OmniSharp that supports most of them (and in the future can support many many more!). Let’s get going then and add our sublime support.

https://github.com/moonrabbit/OmniSharpSublime is the project

So to install:

goto your packages directory (linux here so it may be different in windows or mac, just look in packages in sublime to find the folder)

cd ~/.config/sublime-text-3/Packages/
git clone git@github.com:moonrabbit/OmniSharpSublime.git
cs OmniSharpSublime
git submodule update --init 

Now you have gotten all the needed files. The next thing we will need to do is build OmniSharp

./build.sh

Now edit your project file for sublime and add at the root level

 "solution_file": "./EventStore.sln"

Remember the path is relative from your project file! restart sublime if its running. Try typing out variable names and you will see you have intellisense. If you hit a . you will notice that it does not come up :( by default the auto complete keystroke is alt + / you can remap this to ctrl+space if you want by editing your keymap in sublime.

Want to go to definition of a method? Try f12 by default (again can be remapped its up to you!)

In the next post we will recap everything that we have done so far!

Sublime is Sublime 11

So we are now into the 11th post. There are only two more to go after this one. One with some more functionality and one as a summary. In the last post we installed OpenIDE and showed the very basics of its functionality, adding a file to an existing project.

OpenIDE can do much more than this. It has most support for sln/prj that you will need. Let’s start by making a new project.

greg@goblin:~/src/foo$ oi create console src/HelloConsole
Created src/HelloConsole
greg@goblin:~/src/foo$ ls
src
greg@goblin:~/src/foo/src$ ls
HelloConsole.csproj  Program.cs  Properties

This will create a proejct from a template. The following are the available templates (listed from help).

	create : Uses the create template to create what ever project related specified by the template
		console : Creates a new C# console application
			ITEM_NAME : The name of the Project/Item to create
		library : Creates a new C# library project
			ITEM_NAME : The name of the Project/Item to create
		service : Creates a new C# windows service
			ITEM_NAME : The name of the Project/Item to create

You could remove Program.cs with oi deletefile foo/Program.cs if you wanted and it would also be removed from the project as well.

You can create your own templates as well they are just scripts. This applies to both new items and project templates. If for example you wanted to make a custom item for a new item (say a custom xunit testfixture).

Go to your OpenIDE release. cd .OpenIDE/languages/C#-files/

You will see here there is create and new. These hold the templates for the create and new commands they are implemented as python but can be scripted in any language

As an example here is the template for a new interface

#!/usr/bin/env python
import sys

if __name__ == "__main__":
	if sys.argv[1] == 'get_file_extension':
		print(".cs")
	elif sys.argv[1] == 'get_position':
		print("6|3")
	elif sys.argv[1] == 'get_definition':
		print("Creates an new C# interface")
	else:
		classname = sys.argv[1]
		namespace = sys.argv[2]
		parameterfile = sys.argv[3]
		print("using System;")
		print("")
		print("namespace " + namespace)
		print("{")
		print("	interface " + classname)
		print("	{")
		print("	}")
		print("}")

and here is the template for a new Console Application.

#!/usr/bin/env python
import sys
from files.copydir import copy as copydir

if __name__ == "__main__":
	if sys.argv[1] == 'get_file':
		print("Program.cs")
	elif sys.argv[1] == 'get_position':
		print("8|3")
	elif sys.argv[1] == 'get_definition':
		print("Creates a new C# console application")
	else:
		copydir("console", sys.argv[1])

There is still much that can be added to OpenIDE (and it does a ton of other things we have not covered). But in general it can get you around the issues of dealing with project and solution files including references.

Sublime is Sublime 10

Ok we have been moving right along through the sublime features and getting setup for .net development. I have been half saving the next few posts deliberately until the end as they will be covering the largest arguments I hear against the use of other editors than VS when dealing with .NET code.

But my team uses Visual Studio, I can’t just give up on using project/solution files and use some hipster editor.

This has for a long time been the single largest hurdle in using non-VS editors. If you want to get an idea of how bad it is and you have been following along the posts, try adding a file to a project or reference another project in Sublime. Ouch manually editing project files. How do you know that your manual edit will work when opened in Visual Studio?

To be fair even if it took you 15 seconds per file/reference that you added in Sublime the overall time on the project would be minimal but it is a serious pain in the ass. Nothing makes you feel slower than having to manually edit xml that was automatically done for you previously.

To get some of this functionality we will install a new tool though a whole new tool is not really needed for this. It could be done with some basic shell scripts. The tool is OpenIDE by @ackenpacken. OpenIDE does a whole lot more than what we need it to. I have been chatting with him recently about maybe making it more modular hell even Mighty Moose is contained within it as of now.

OpenIDE supports some of the generic things you would want when working with .NET code. The ability to edit project/solution files. The ability to handle templating for new additions. Reference management. There are also some other tools out there as well such as omnisharp https://github.com/nosami/Omnisharp but I fear all of them are too complex and not modular enough as there hasn’t been much of a push for that kind of tooling. Part of this post series is to help mold demand for these kinds of tools.

Now for OpenIDE install. You can grab the sources for OpenIDE here https://github.com/continuoustests/OpenIDE. Svein has recently added a binary repository here https://github.com/continuoustests/OpenIDE.binaries.git. Pull the binaries repository or build from sources. Put the output into your $PATH. OpenIDE also comes with bash completion if you want to install it which can help greatly! Now you are good to start.

Let’s make sure OpenIDE works: oi

You should get help.

oi package install C-Sharp

In the root of your project type oi init C#

Now oi is setup and ready to go. From the command line let’s try

greg@goblin:~/src/EventStore/src/EventStore$ oi new class esquery/bar
Created class esquery.bar
Full path /home/greg/src/EventStore/src/EventStore/esquery/bar.cs

Note that I did not put bar.cs just esquery/bar and yes you get tab completion on this.

If I now look at what changed.

greg@goblin:~/src/EventStore/src/EventStore$ git status
# On branch dev
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   esquery/esquery.csproj
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	esquery/bar.cs
greg@goblin:~/src/EventStore/src/EventStore$ git diff esquery/esquery.csproj
diff --git a/src/EventStore/esquery/esquery.csproj b/src/EventStore/esquery/esqu
index dec282f..9f3c95f 100644
--- a/src/EventStore/esquery/esquery.csproj
+++ b/src/EventStore/esquery/esquery.csproj
@@ -84,6 +84,7 @@
     <Compile Include="CommandProcessor.cs" />
     <Compile Include="Program.cs" />
     <Compile Include="Properties\AssemblyInfo.cs" />
+    <Compile Include="bar.cs" />
   </ItemGroup>
   <ItemGroup>
     <None Include="app.config" />

You can also run this command directly inside of sublime. Just use ctrl+shift+c and type in your command. This is just the beginning though. OpenIDE and such tools can support most of your integration with things like project/solution files.

Sublime is Sublime 9

Yesterday I took a survey of .net developers as I was curious how most are working. In particular I was wondering what % of time people actually work on .NET code vs working on code such as html or javascript and in particular whether they were using VS for html and javascript development.

I would love to put all of the raw data here but apparently surveymonkey wants 25/euros per month to actually export the results of a survey which sucks. Need to find a new survey tool.

What I found was a few groups of .NET developers one that was quite interesting for me answered the questions in this way:

Q1: What % of your time are you actively coding at work?
80
Q2: What % of your coding time is in .NET code (C#/F#/VB.NET/etc)
30
Q3: What % of your coding time is in HTML?
30
Q4: What % of your coding time is in JavaScript?
40
Q5: Do you use Visual Studio for javascript/html?

There were a huge number of developers who spent 40-70% of their coding time in Visual Studio not working on C# code but instead working on javascript and html code. For these developers in particular I point you to the html and javascript support in sublime without comment just watch a few videos, getting setup is just installing a few packages through your package manager (discussed previously in this series)

 

cookie-monster

WebInspector – http://sokolovstas.github.io/SublimeWebInspector/ seriously watch the video
Emmet – http://emmet.io/ formerly known as zencoding.
Code Intel – http://sublimecodeintel.github.io/SublimeCodeIntel/ go to definition etc support

Tomorrow we will continue the series getting into .net specific stuff and fixing some areas we are weak in now. How do you add a reference?

Anyone used any of these plugins? Please share your experiences in comments.

Sublime is Sublime 8

In this post we will look at how to become a bit more efficient in our sublime setup without adding lots of new tools or complexity. In particular I don’t like running my builds and my tests manually (and we will look at some varying options for this later). In this post however we will look at a dead simple method for doing it.

Let’s take a look at the full script I briefly wrote that does all of this.

#!/bin/sh
notify-send "Auto Started" "" -i $PWD/images/circleWIN.png -t 1000
while true
do
      if [ ! -z $(inotifywait --recursive -qre modify --format "%w%f" ./ | grep -v -f includes) ]
          then
	  sleep .25
          /opt/mono/bin/xbuild src/EventStore/EventStore.sln /p:Configuration=Release /verbosity:0 /nologo
          if [ $? -eq 0 ]
            then
               ./run_tests.sh -m /opt/mono -x LongRunning 
                if [ $? -eq 0 ]
                then
                      notify-send "Passed" "Tests Passed" -i $PWD/images/circleWIN.png -t 1000
                else
                      notify-send "Failed" "Tests Failed" -i $PWD/images/circleFAIL.png -t 1000
                fi
            else
                notify-send "Failed" "Build Failed" -i $PWD/images/circleFAIL.png -t 1000
          fi 
      fi
done

So what this does is wait for file changes in the directory structure (notify wait). It then checks to see if the file returned matches an include filter eg “*.cs” if it does it will then build the code followed by running the tests. When you get a failure/success you will receive a notification including a red/green circle in xwindows (regardless of if you are still in sublime). If you want to see run_tests.sh you can view it here https://github.com/EventStore/EventStore/blob/dev/run_tests.sh its a shell script in EventStore to ease in the running of tests on linux.

This is a great showing of the “unix philosophy” of small tools being composed together to get something larger. It was not hard to make this but it provides 80% of the value of autotest.net and is vastly more configurable if you understand shell script (want an exclude vs an include just edit the script).

There is no fancy installer. There is no visual studio integration to do (I will never do that ever again). On EventStore source base which is by all means not a small source base (roughly 2000 tests which are run locally) the entire cycle is 15-20 seconds on my machine. Although MightyMoose could have gotten this to around 5 which would be even better for simplicity this wins out.

One problem that I ran into had to deal with nunit-console when you have ignored tests there is no way to get it to not print out all the ignored tests (which it does of course last! so you can’t see your failures). To work around that I wrote a small xslt to format its results

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0"	
                exclude-result-prefixes="msxsl"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
		xmlns:msxsl="urn:schemas-microsoft-com:xslt">
<xsl:output method="html" indent="yes" />
<xsl:template match="test-results">
			<xsl:variable name="failedtests.list" select="//test-case[@success='False']"/>
			<xsl:variable name="total" select="@total + @not-run" />
   			<xsl:variable name="success" select="@total - @failures - @errors" />
          	Total  : <xsl:value-of select="$total" /> Success: <xsl:value-of select="$success" /> Failed : <xsl:value-of select="@failures" /> Not Run: <xsl:value-of select="@not-run" />
			<xsl:apply-templates select="$failedtests.list"/>
</xsl:template>
<xsl:template match="test-case[@success='False']">
*************************************************************************************************	
	<xsl:value-of select="@name"/>
	<xsl:value-of select="failure/message"/>
	<xsl:value-of select="failure/stack-trace" />
</xsl:template>
</xsl:stylesheet>

This will give you a printout that looks like this:

          	Total  : 1748 Success: 1699 Failed : 1 Not Run: 48
*************************************************************************************************
	EventStore.Core.Tests.Services.UserManagementService.password_change_notification_reader.when_notification_has_been_written.publishes_reset_password_cache  Expected: 1
  But was:  0
at EventStore.Core.Tests.Services.UserManagementService.password_change_notification_reader.when_notification_has_been_written.publishes_reset_password_cache () [0x00000] in &lt;filename unknown&gt;:0

Again we have not done anything complicated here we are just composing our exists tools. I should add that you can easily make those links clickable back into sublime to go to a line based on stack trace but you either need to switch your terminal from gnometerminal or you need to add one line of code and custom build gnome terminal which may be a bit much for some people :)

As a quick exercise could you write a script like above for javascript?

Sublime is Sublime 7

So as we have come along the path our sublime instance is starting to feel quite a bit more customized than it was before. We can now build, have git/github integration, and of course have awesome command line integration but what about running tests?

We will start with the simplest way of running tests. This may not work for all of your scenarios and does somewhat require you to have thought about things in advance.

If you for instance started off from the beginning with resharper as a test runner you are probably have a test suite that will be difficult to use. Likely it is all in one assembly regardless of whether its a unit test or an integration test. Quite likely the tests have large numbers of dependencies. And to be frank your unit tests probably don’t run properly in linux anyways. Yeah this kind of sucks but can be fixed.

To start with there is no integrated right click->debug test in sublime. I find however that this is not something I missed. Maybe its having worked for a few years building embedded systems where there was no debugger (except dumping as hex pieces of memory).

Let’s talk about the simplest way to run tests from sublime and the general workflow. Normally when working with sublime I have multiple test suites either separated by category or by assembly. I have unit tests which are quick to run and then all other tests. I rarely if ever run the other tests locally. Instead I let a CI server run those. My unit tests should be run in and this would give you a new buffer with the output of your tests running. For a huge portion of code you will be working on this is *good enough*. If you have a 10 second cycle on tests running it is completely reasonable. eg

ctrl+shift+c run >

where I have a script called run that runs nunit in the way I want.

There are however some other things you can do if you want to be more fancy. The first is you could assign the running of the script to a key in sublime. This is actually quite easy any command can be assigned to any key by editing your keymap http://sublimetext.info/docs/en/reference/key_bindings.html

Another cool option that is quite easy to enable to get a notification from linux on the status of your build/test run. This can be done trivial using notify-send in your small script above based on return values from processes.

Try:
notify-send ‘Hello world!’ ‘This is an example notification.’ –icon=dialog-information

You can even change the icons to say red/green circles for the notification.

Now if your tests are more than 10 seconds. The first strategy would be to start putting them into categories that make sense to you. This way you can only run tests within a given category. This step alone will normally be good enough to get your test runs back under 10 seconds (I prefer 1-2 seconds). If this is not then perhaps your code base has become too dependent on the tool (15 categories 10 seconds each…). One thing you will notice is that your tests in general will run far faster than through tools like VS/R# when being run from the command line.

OK so now we have basic unit test integration. There are still a lot of problems with .NET code. In the next post we will look at what some of these are and how to deal with them.

Follow

Get every new post delivered to your Inbox.

Join 8,257 other followers