If your team is like my team, you're trying to find ways to automate everything. Perhaps you're also using watir-webdriver to run browser based tests with Cucumber. This works great locally, but when you try to run your tests on a CI environment you start to run into some very strange problems with Firefox testing. You start to get errors around not being able to find the h1 tag on your page. You look at the screen shot you've generated using a method similar to this, and you find that your screenshot shows the text that contains the h1. In fact, your error dumped html file also shows the h1 tag:
You've run into selenium issue 2099. It appears what happens with this error is that the test is executing before the page has fully loaded, causing it to fail. Old school race conditions. Nice.
So how do we get around this? There's been a few postings on how to handle this with Capybara, but what about tools that use selenium-webdriver more directly? Here's a monkey patch that hacks over the Ruby bindings to enable tools like watir-webdriver to retry. This patch will retry the find until selenium-webdriver receives an error that is not related to the strange InvalidSelectorError that it gets when the Firefox isn't done loading the page. I just stuff this file into the support directory in a Cucumber suite.
Since this is hacky and a monkey patch, I'm very keen on ideas on how to improve this implementation, since my Ruby skills are very raw.
Friday, October 21, 2011
Thursday, September 22, 2011
Integrating jasmine-maven-plugin With Your Legacy Enterprisey App
Let's say you're working on enhancing an old legacy app with javascript. Doesn't really matter what sort of app, just legacy, with an old source tree and an old deployment style. How can you integrate jasmine within that app and have it also integrate with your new fangled CI environment? What if you've also been asked to use maven since it's "The way forward" for your organization? How can you accomplish this?
Well, there's the jasmine-maven-plugin, which is all sorts of awesome, but because you have a legacy file structure there's no clear way on how to hook it into your legacy app. But when there's a will there's a way.
There are several problems that need to be tackled here. Let's go over them bit by bit.
First, how do we get jasmine-maven-plugin to see our legacy app?
Let's hope that you can create your project somewhere "near" the legacy app in the SCM system. In this example, I'm going to assume that we have a tree structure that looks like this:
Pretty simple, we'll put our specs inside jasmine-tests\src\test\javascript. We'll put any jasmine required javascript inside jasmine-tests\src\test\javascript\lib. We just need to figure out how to tell jasmine-maven-plugin to look at our legacy app for our scripts. Fortunately, it's configurable. Let's look at a snippet of the pom.xml that shows how you would configure the plugin.
In this configuration, we're telling the jasmine-maven-plugin to look outside of our project structure for our javascript files. It even preloads our new fangled jquery thing that we've introduced into our legacy application from our legacy.js.src property. The key to this is that we've kept the configuration of the plugin outside of our execution configuration. When jasmine-maven-plugin runs, maven will actually fork a process to generate the manual runner. It will not receive the configuration settings if you're using an older maven like 2.0.11. I do not know if 2.2.1 or 3.x change this. By placing the configuration at the top level of the plugin, we fix that concern.
At this point, we actually have a plugin that can execute on our system if we've checked out our entire source tree. It should also execute in your CI environment.
However, some teams, like to check things out in different directory structures than what's in the SCM tree due to their tools (like Eclipse). Although your team is excited, they've also noticed that the ManualTestRunner is deprecated. When they try to run jasmine:bdd, they find that nothing gets picked up from the legacy app because the jetty runs from the root of your jasmine-tests directory. It can't see things underneath it. Normally you'd create a symbolic link, but...you're on Windows. Windows XP. Remember, I said Enterprisey app, not just legacy.
How can you fix the process so you can run jasmine:bdd like good sensible developers? Glad you asked. Maven profiles to the rescue. And, a neat plugin that lets you build symbolic links on crazy things like Windows XP.
Let's look at the profile:
The maven-junction-plugin actually builds symbolic links in both unix and windows. We redefine the location of our legacy.js.src to be a temporary directory inside the jasmine-tests project. The checkedout.legacy.js.src points to where our application has been checked out locally. The plugin is executed in the initialize phase so it occurs before jasmine. And initialize is supposed to setup directory structures, and the like, so it makes sense.
Since we've redefined the legacy.js.src, jasmine:bdd will simply see our files as if they are inside the project, and the jetty will be able to serve them properly.
Our final pom looks like this:
Hopefully this is helpful for you.
Well, there's the jasmine-maven-plugin, which is all sorts of awesome, but because you have a legacy file structure there's no clear way on how to hook it into your legacy app. But when there's a will there's a way.
There are several problems that need to be tackled here. Let's go over them bit by bit.
First, how do we get jasmine-maven-plugin to see our legacy app?
Let's hope that you can create your project somewhere "near" the legacy app in the SCM system. In this example, I'm going to assume that we have a tree structure that looks like this:
Pretty simple, we'll put our specs inside jasmine-tests\src\test\javascript. We'll put any jasmine required javascript inside jasmine-tests\src\test\javascript\lib. We just need to figure out how to tell jasmine-maven-plugin to look at our legacy app for our scripts. Fortunately, it's configurable. Let's look at a snippet of the pom.xml that shows how you would configure the plugin.
In this configuration, we're telling the jasmine-maven-plugin to look outside of our project structure for our javascript files. It even preloads our new fangled jquery thing that we've introduced into our legacy application from our legacy.js.src property. The key to this is that we've kept the configuration of the plugin outside of our execution configuration. When jasmine-maven-plugin runs, maven will actually fork a process to generate the manual runner. It will not receive the configuration settings if you're using an older maven like 2.0.11. I do not know if 2.2.1 or 3.x change this. By placing the configuration at the top level of the plugin, we fix that concern.
At this point, we actually have a plugin that can execute on our system if we've checked out our entire source tree. It should also execute in your CI environment.
However, some teams, like to check things out in different directory structures than what's in the SCM tree due to their tools (like Eclipse). Although your team is excited, they've also noticed that the ManualTestRunner is deprecated. When they try to run jasmine:bdd, they find that nothing gets picked up from the legacy app because the jetty runs from the root of your jasmine-tests directory. It can't see things underneath it. Normally you'd create a symbolic link, but...you're on Windows. Windows XP. Remember, I said Enterprisey app, not just legacy.
How can you fix the process so you can run jasmine:bdd like good sensible developers? Glad you asked. Maven profiles to the rescue. And, a neat plugin that lets you build symbolic links on crazy things like Windows XP.
Let's look at the profile:
The maven-junction-plugin actually builds symbolic links in both unix and windows. We redefine the location of our legacy.js.src to be a temporary directory inside the jasmine-tests project. The checkedout.legacy.js.src points to where our application has been checked out locally. The plugin is executed in the initialize phase so it occurs before jasmine. And initialize is supposed to setup directory structures, and the like, so it makes sense.
Since we've redefined the legacy.js.src, jasmine:bdd will simply see our files as if they are inside the project, and the jetty will be able to serve them properly.
Our final pom looks like this:
Hopefully this is helpful for you.
Monday, August 01, 2011
The Problem with PageObjects, why PageParts are needed
Over the past year or so, several big dogs in the automated web testing community have been pushing hard with the concept of PageObjects. This is a great thing. As such we now have at least two different page object frameworks to use with Watir-Webdriver thanks to the efforts of Alister Scott and Cheezy.
However, the page object frameworks have placed an item on the backburner. I'm not sure why it is, but the concept of repeating items inside of a page has been ignored at this point. Let's say you have a website like Google Finance. The list of news items is a great example of a what I'm calling a PagePart.
PageParts are items that repeat themselves on a page in terms of content types. They repeat with different content depending upon the user who is logged in, but they all look the same semantically. Here's what a Google Finance News item looks like:
There are four of these items wrapped in a div with market-news-stream as it's id.
Without PageParts you might do this:
So how do we identify these items in the tests? When I first started tackling this sort of problem on my current project I had pass through methods inside my PageObject that went to a simple PagePart. This worked, but didn't give me the cool features of the PageObject frameworks, and wasn't that the whole point of adopting the framework?
Wow, this is more complex than not using page parts.
So, how exactly is this better? It's better when you convert your page parts to use the PageObject framework. Caveat, I do not know how you would do this in Cheezy's PageObjects gem, but it works just fine in the watir-page-helper gem:
Every PagePart extends the BasePart. You'll select your container div in your PagePart object using the part_div method. Obviously this could be any HTML element, but it's most likely to be a div. Since the BasePart will redefine @browser with the div, we'll be able to use the cool shorthand, rather than define every method we might need:
The example may seem a bit contrived, but, maybe you're not working with an interface as clean as Google Finance. If you are, you might get some benefit out of using PageParts.
However, the page object frameworks have placed an item on the backburner. I'm not sure why it is, but the concept of repeating items inside of a page has been ignored at this point. Let's say you have a website like Google Finance. The list of news items is a great example of a what I'm calling a PagePart.
PageParts are items that repeat themselves on a page in terms of content types. They repeat with different content depending upon the user who is logged in, but they all look the same semantically. Here's what a Google Finance News item looks like:
There are four of these items wrapped in a div with market-news-stream as it's id.
Without PageParts you might do this:
So how do we identify these items in the tests? When I first started tackling this sort of problem on my current project I had pass through methods inside my PageObject that went to a simple PagePart. This worked, but didn't give me the cool features of the PageObject frameworks, and wasn't that the whole point of adopting the framework?
Wow, this is more complex than not using page parts.
So, how exactly is this better? It's better when you convert your page parts to use the PageObject framework. Caveat, I do not know how you would do this in Cheezy's PageObjects gem, but it works just fine in the watir-page-helper gem:
Every PagePart extends the BasePart. You'll select your container div in your PagePart object using the part_div method. Obviously this could be any HTML element, but it's most likely to be a div. Since the BasePart will redefine @browser with the div, we'll be able to use the cool shorthand, rather than define every method we might need:
The example may seem a bit contrived, but, maybe you're not working with an interface as clean as Google Finance. If you are, you might get some benefit out of using PageParts.
Wednesday, April 13, 2011
Making Your Cukes More Awesomer With Screenshots
In my last post about Cucumber, I alluded to using watir-webdriver instead of using native watir. There was one very surprising advantage I discovered with watir-webdriver's use of selenium as a backend: Awesome Screenshot API.
Here's how we setup our Cucumber suite to take screenshots (and html/txt "screenshots") using watir-webdriver and celerity. First, set your tests up to write an error when a test fails:
In this hook, we're passing in an environment, the scenario and the browser object we setup in the prior post. The environment is really handy since QA folks might be running tests against different regions and would want to pass that information back to the developers to recreate problems. The TEST_SERVER environment variable is already used to pick this during test startup.
The ErrorWriter module is next.
Several things are going on here. The write errors message is simple enough. It creates a directory for the errors to be written, and dumps the context of the browser's text and html representations to file. What's more interesting are the scenario_name and file_name methods.
The scenario_name method does some interesting things. We want to name our files by the name of the scenario that fails. In the else clause, you see how dashes and white space are replaced by a single white space character. But, in the if clause, we're testing to see if we have an ExampleRow. These are the Examples in a Cucumber Scenario Outline. If the scenario is an ExampleRow, we lose the name of the Scenario itself. Fortunately, the ExampleRow knows what ScenarioOutline it belongs to. We can get the name of the ScenarioOutline, perform the same transform on the name, and then append the example that is actually running to it's name.
The file_name method simply concatenates the region, time of failure and the scenario name together for the filename to be written to.
This can result in some very long file names, however, their descriptiveness wins overall. You don't have to open a file to find out what failed. A simple reading of the directory containing the files lets anyone find the test that they need to examine quickly.
The write_errors method did one other thing. Call capture_screenshot to grab an image of the browser. When using Watir, you had to do some fancy things with the Win32 API to make the capture work. We had a lot of issues using it with our QA folks who had SnagIT installed. So how refreshing it was to have watir-webdriver just work with so little code:
We have another little trick, where in our prior example, we setup a WEBDRIVER variable to determine if we could take a screenshot. If we're running on Celerity, no harm, no foul, but no screenshot.
Granted, it's really selenium that we're using here, but we get really nice and small png files, which only contain the contents of the browser. No title bar, no toolbars, nothing. Just the contents of the window. Our only issue is that IE causes problems on our CI box, but just having Firefox able to run there is an outstanding step forward.
Hopefully this is helpful, please let me know if it is.
Here's how we setup our Cucumber suite to take screenshots (and html/txt "screenshots") using watir-webdriver and celerity. First, set your tests up to write an error when a test fails:
In this hook, we're passing in an environment, the scenario and the browser object we setup in the prior post. The environment is really handy since QA folks might be running tests against different regions and would want to pass that information back to the developers to recreate problems. The TEST_SERVER environment variable is already used to pick this during test startup.
The ErrorWriter module is next.
Several things are going on here. The write errors message is simple enough. It creates a directory for the errors to be written, and dumps the context of the browser's text and html representations to file. What's more interesting are the scenario_name and file_name methods.
The scenario_name method does some interesting things. We want to name our files by the name of the scenario that fails. In the else clause, you see how dashes and white space are replaced by a single white space character. But, in the if clause, we're testing to see if we have an ExampleRow. These are the Examples in a Cucumber Scenario Outline. If the scenario is an ExampleRow, we lose the name of the Scenario itself. Fortunately, the ExampleRow knows what ScenarioOutline it belongs to. We can get the name of the ScenarioOutline, perform the same transform on the name, and then append the example that is actually running to it's name.
The file_name method simply concatenates the region, time of failure and the scenario name together for the filename to be written to.
This can result in some very long file names, however, their descriptiveness wins overall. You don't have to open a file to find out what failed. A simple reading of the directory containing the files lets anyone find the test that they need to examine quickly.
The write_errors method did one other thing. Call capture_screenshot to grab an image of the browser. When using Watir, you had to do some fancy things with the Win32 API to make the capture work. We had a lot of issues using it with our QA folks who had SnagIT installed. So how refreshing it was to have watir-webdriver just work with so little code:
We have another little trick, where in our prior example, we setup a WEBDRIVER variable to determine if we could take a screenshot. If we're running on Celerity, no harm, no foul, but no screenshot.
Granted, it's really selenium that we're using here, but we get really nice and small png files, which only contain the contents of the browser. No title bar, no toolbars, nothing. Just the contents of the window. Our only issue is that IE causes problems on our CI box, but just having Firefox able to run there is an outstanding step forward.
Hopefully this is helpful, please let me know if it is.
Wednesday, April 06, 2011
Cucumber with Celerity and Watir-Webdriver
Earlier this year, I had the problem of trying to enable our QA staff to run our current suite of Celerity driven Cucumber tests through the actual browsers that we were using. One obvious solution is to use various flavors of Watir.
The main concern with this solution is that Waitr required native ruby instead of jruby. Not a huge deal, but still an extra annoyance for folks in a java shop. Requiring QA to switch between jruby and watir wasn't ideal. Additionally, taking snapshots of the screen during a failed test was far from trivial.
So instead, I embarked on a quest to use watir-webdriver and Celerity together. The advantage here is that watir-webdriver, through selenium-webdriver, can talk to browsers and runs on jruby. This combination was alluded to on several blog sites that I had found, but I never quite found something that put it all together for me. Eventually, i got it working. Hopefully this will at least get you about 90% of the way there.
The first step would be adding watir-webdriver to your Gemfile:
Following that we create a browser.rb file that determines what sort of browser you are running. I placed this in the support folder in the Cucumber project. One thing to remember about this, is that you could also throw this method into your env.rb.
What's going on here? We're using the environment variable BROWSER to determine if we want to run something in a browser. We then use another environment variable to determine which browser to use. We even have an environment variable to determine what Firefox we want to use, which is a nice feature of selenium-webdriver. At the end, Browser contains the class of browser we want to instantiate for tests, and BROWSER_TYPE contains the actual type of browser to instantiate for watir-webdriver.
One other thing to note, when we use Celerity, we're setting the offset for index based searches to 0. Unlike watir-webdriver, Watir uses 1th based arrays. Celerity, being a wrapper of HtmlUnit, had to translate those arrays from 1 to 0 so that HtmlUnit would run properly. By setting this value, we can use 0th based arrays, just like watir-webdriver does in Celerity.
Next we setup our Cucumber Before block in hooks.rb:
In here, we simply instantiate the object. In my case, we also needed to setup some URLs and turn off SSL for our Celerity testing.
The Rakefile is probably a bit more complex due to our desire to run both headless and browser tests on our CI box, but it is helpful, since it uses the :browser task to setup the BROWSER environment variable.
Finally, to help our QA folks, three batch files to execute the tests.
Hopefully you'll find this useful. Please let me know if you do.
The main concern with this solution is that Waitr required native ruby instead of jruby. Not a huge deal, but still an extra annoyance for folks in a java shop. Requiring QA to switch between jruby and watir wasn't ideal. Additionally, taking snapshots of the screen during a failed test was far from trivial.
So instead, I embarked on a quest to use watir-webdriver and Celerity together. The advantage here is that watir-webdriver, through selenium-webdriver, can talk to browsers and runs on jruby. This combination was alluded to on several blog sites that I had found, but I never quite found something that put it all together for me. Eventually, i got it working. Hopefully this will at least get you about 90% of the way there.
The first step would be adding watir-webdriver to your Gemfile:
Following that we create a browser.rb file that determines what sort of browser you are running. I placed this in the support folder in the Cucumber project. One thing to remember about this, is that you could also throw this method into your env.rb.
What's going on here? We're using the environment variable BROWSER to determine if we want to run something in a browser. We then use another environment variable to determine which browser to use. We even have an environment variable to determine what Firefox we want to use, which is a nice feature of selenium-webdriver. At the end, Browser contains the class of browser we want to instantiate for tests, and BROWSER_TYPE contains the actual type of browser to instantiate for watir-webdriver.
One other thing to note, when we use Celerity, we're setting the offset for index based searches to 0. Unlike watir-webdriver, Watir uses 1th based arrays. Celerity, being a wrapper of HtmlUnit, had to translate those arrays from 1 to 0 so that HtmlUnit would run properly. By setting this value, we can use 0th based arrays, just like watir-webdriver does in Celerity.
Next we setup our Cucumber Before block in hooks.rb:
In here, we simply instantiate the object. In my case, we also needed to setup some URLs and turn off SSL for our Celerity testing.
The Rakefile is probably a bit more complex due to our desire to run both headless and browser tests on our CI box, but it is helpful, since it uses the :browser task to setup the BROWSER environment variable.
Finally, to help our QA folks, three batch files to execute the tests.
Hopefully you'll find this useful. Please let me know if you do.
Monday, March 28, 2011
My First Code Retreat...in Prison (Coding in the Clink III)
This weekend, I had the privilege of being part of a Code Retreat in Prison. Being that I haven't been to a Code Retreat...period, it might seem odd that I would choose The Clink as the venue for my first Code Retreat.
Along those lines, one of the main questions I got from folks on the outside is why? For me, this was a fairly simpler question, but the answer is a bit more complex than I let on.
The first reason is, I knew Dan from my time working at CheckFree. I still remember, and tell stories, of Dan and I being in the same HR orientation together. Dan asked some very good questions surrounding code ownership, and inventions, and who owned inventions made at on our own time at home. The poor HR staff hadn't a clue how to answer Dan's questions. I suspect now they do, since they were excellent points.
The second reason is I had the pleasure of working with Joel, while at Nationwide. Joel became involved in working in the program on Tuesday nights at MCI. I found the stories of his experience fascinating and inspiring. The fact that Dan was involved, made it all the more interesting. The third reason, however, is where it gets more complex.
Growing up, I attended mass on a weekly basis. When you attend mass on a weekly basis you get to experience different style homilies from different priests. Each one has their own preffered style. Our pastor tended to simplify things such that a 5 year old could understand it. Other priests essentially took the guidance provided by the church and recited it back such that it.felt like a lecture. The best homilies, like most good programming conference talks, were filled with stories that made the subject matter relevant to your life today.
My favorite homilies contained stories of when the father visited inmates in prison. There was nothing more inspiring than these stories of redemption and understanding. In fact, it was very obvious, that some of these inmates had achieved a better understanding and faith than I had as a questioning teenager. I found myself wishing I could do something like that for something I truly understood. Fast forward twenty years, and I have that opportunity. I couldn't teach an inmate about faith, but I could bring a skill that I had cultivated over time and maybe do my part. How could I pass this up?
So on Sunday morning I took the drive up to Marion Correctional Institution. Myself, and seven other "outsiders" went into the prison and walked to the Life Line center. One thing that became very apparent during the walk was that you were in prison. You weren't in a place off to the side of the prison, a visitors area or anything like that. You walked right past the kitchen and dining hall. Which was a very good thing actually. Kairos was also there this weekend. There is nothing that will calm your nerves than hearing a hundred or so men, throatily singing gospel songs as part of that program in the dining hall.
Once we made it to Life Line, we were greeted by the inmates who were very cordial and polite. We had short introductions and offered our skill levels in Java. We then decided to have each outsider sit at a workstation and have the prisoners pair or triple with each of us. I had the opportunity to triple with a beginner and a more intermediate to senior level inmate on our first effort on a Tennis Game Kata.
We made a few obvious mistakes. Trying to cram an enum into the problem wasn't ideal and sidetracked us. We probably should have ping ponged a bit more in our pairing style, as what ended up happening was our beginner programmer drove while myself and the more senior inmate acted in a coaching role. Not that there's anything wrong with this approach, but we didn't make it very far in our effort to solve the problem. This, of course, is OK, since you're going to toss the code and start over for round 2 with different pairs.
We then had the opportunity to have lunch, which meant eating with the population in the dining hall. With four people to a table, we intemingled and chatted over a brunch. Brunch consisted of sausage, a hashbrown, an english muffin with jelly, some sort of bran flakes, scrambled eggs, a salad and a banana. Seems like a lot of food until you realize, you don't get to snack on the inside. One of the more interesting experiences was having a random inmate come and ask me if I wanted my banana. Once I discovered that was fine, I gave it to him. It's apparently quite common.
During lunch, we had an interesting discussion about programming and its utility on the outside of being able to earn a living. This was a popular refrain. Could you get a job in programming? I think some of these guys had seen internal programs go away due to foreign competition and they wanted to know if there was a future in it. Yes, the furniture building program had been halted since the state could get product cheaper overseas. I think the choice of the prison to shift resources into technology is a very good move, it will help the inmates upon release regardless of programming's place.
What was also interesting was learning that some of the guys had already programmed in prison before. I heard a story of guys working for a client in Arizona. They ran into issues, however, when the client's programmer had a complex of working with programmers in prison. Sounds similar to folks having concerns with working with offshore developers, doesn't it? Probably the most interesting thing about this situation was that all communication about the code was done over telephone. No email, no instant message, no internet. Just telephone and books. That's tough!
After lunch I had the opportunity to see a 90 second space battle video that an inmate rendered using Blender and some other graphics tools. It was darn impressive. There was serious skill involved in making the video.
After the video we switched pairs, and I ended up with two fellows who I would classify as intermediate to advanced in programming skills. We were able to solve the problem through rotational ping pong pairing. Probably the most interesting thing about this was towards the end. We believed we were done, until we read the final step of the kata which told us that the program should ignore all points after a game is won. Writing tests for that feature drew up a bunch of tiny hidden bugs which gave me the opportunity to talk about self documenting code and extracting methods to handle those problems. Which, one of my triple knew quite well...probably better than I did, but was a bit too modest.
One of the things that occurred to me after the second pairing was how skilled some of these programmers were. I can honestly say that some of these guys had more skill than folks who I've been involved in interviews with on the outside. I believe a lot of it has to do with rote memorization. They have no Internet. Only books, Eclipse and each other. Any plugins and tools have to be brought in from the outside. They don't have StackOverflow to ask questions. Just each other, books, and whatever hints the IDE provides. So unlike most developers, who have adapted lazy instantiation of skills in order to adapt quickly to different technologies, these guys have had to battle harden the skills that they have. The other thing that surprised me, is how well they pair. One thing you might not expect in a prison situation is the ability to share information, reduce your ego, and pass the keys back and forth. Yet, they did it without concern. Of the advanced programmers, I had the luxury of pairing with, I saw folks who would have no problem fitting in with an Agile team. I know they could pass an audition, if they could get one on the outside.
So when I reflect on my experience, it makes me take a step back, and think about my own ego when I pair. Am I giving up the keyboard enough? Am I ping ponging enough? Am I letting a junior developer work through their problems enough, or do they need to shoot themselves in the foot a few times first? It was great reflection for me in handling my professional life.
One final note. The guys in here have made mistakes. Some of these guys have made really big mistakes. I didn't ask, but one of the inmates I paired with offered that he had been in the system for almost two decades. You can draw your own conclusions from that information. He had serious questions on if learning Java could help him when he got to the outside. I think it can, but as Dan put it at our retrospective dinner, the most important thing to learn is pairing and learning to work well with others. I believe the folks in this program were trying to do whatever they can to build skills that can help them later, be it on the inside or outside.
The most interesting thing about this experience, is how well it paralleled with the homilies given in my youth. It became obvious to me the older I got, that the priest was trying to say that, these prisoners know more about faith that you understand due to their situation. Similarly, these inmates knew more about programming than many folks I've encountered simply due to their faith that it's a skill that can help them later. They practiced hard, and were better than they would ever care to admit. Faith is something that isn't taught, but arrives, and it was inspiring to see.
I strongly encourage any programmer somewhere near Marion (any by near I mean 2.5 hour drive...we had folks from Ann Arbor show up), to try to make an effort to make it to a Code Retreat there. If you aren't...maybe a prison near you has a programming program you can volunteer in. It will be worth your time.
Along those lines, one of the main questions I got from folks on the outside is why? For me, this was a fairly simpler question, but the answer is a bit more complex than I let on.
The first reason is, I knew Dan from my time working at CheckFree. I still remember, and tell stories, of Dan and I being in the same HR orientation together. Dan asked some very good questions surrounding code ownership, and inventions, and who owned inventions made at on our own time at home. The poor HR staff hadn't a clue how to answer Dan's questions. I suspect now they do, since they were excellent points.
The second reason is I had the pleasure of working with Joel, while at Nationwide. Joel became involved in working in the program on Tuesday nights at MCI. I found the stories of his experience fascinating and inspiring. The fact that Dan was involved, made it all the more interesting. The third reason, however, is where it gets more complex.
Growing up, I attended mass on a weekly basis. When you attend mass on a weekly basis you get to experience different style homilies from different priests. Each one has their own preffered style. Our pastor tended to simplify things such that a 5 year old could understand it. Other priests essentially took the guidance provided by the church and recited it back such that it.felt like a lecture. The best homilies, like most good programming conference talks, were filled with stories that made the subject matter relevant to your life today.
My favorite homilies contained stories of when the father visited inmates in prison. There was nothing more inspiring than these stories of redemption and understanding. In fact, it was very obvious, that some of these inmates had achieved a better understanding and faith than I had as a questioning teenager. I found myself wishing I could do something like that for something I truly understood. Fast forward twenty years, and I have that opportunity. I couldn't teach an inmate about faith, but I could bring a skill that I had cultivated over time and maybe do my part. How could I pass this up?
So on Sunday morning I took the drive up to Marion Correctional Institution. Myself, and seven other "outsiders" went into the prison and walked to the Life Line center. One thing that became very apparent during the walk was that you were in prison. You weren't in a place off to the side of the prison, a visitors area or anything like that. You walked right past the kitchen and dining hall. Which was a very good thing actually. Kairos was also there this weekend. There is nothing that will calm your nerves than hearing a hundred or so men, throatily singing gospel songs as part of that program in the dining hall.
Once we made it to Life Line, we were greeted by the inmates who were very cordial and polite. We had short introductions and offered our skill levels in Java. We then decided to have each outsider sit at a workstation and have the prisoners pair or triple with each of us. I had the opportunity to triple with a beginner and a more intermediate to senior level inmate on our first effort on a Tennis Game Kata.
We made a few obvious mistakes. Trying to cram an enum into the problem wasn't ideal and sidetracked us. We probably should have ping ponged a bit more in our pairing style, as what ended up happening was our beginner programmer drove while myself and the more senior inmate acted in a coaching role. Not that there's anything wrong with this approach, but we didn't make it very far in our effort to solve the problem. This, of course, is OK, since you're going to toss the code and start over for round 2 with different pairs.
We then had the opportunity to have lunch, which meant eating with the population in the dining hall. With four people to a table, we intemingled and chatted over a brunch. Brunch consisted of sausage, a hashbrown, an english muffin with jelly, some sort of bran flakes, scrambled eggs, a salad and a banana. Seems like a lot of food until you realize, you don't get to snack on the inside. One of the more interesting experiences was having a random inmate come and ask me if I wanted my banana. Once I discovered that was fine, I gave it to him. It's apparently quite common.
During lunch, we had an interesting discussion about programming and its utility on the outside of being able to earn a living. This was a popular refrain. Could you get a job in programming? I think some of these guys had seen internal programs go away due to foreign competition and they wanted to know if there was a future in it. Yes, the furniture building program had been halted since the state could get product cheaper overseas. I think the choice of the prison to shift resources into technology is a very good move, it will help the inmates upon release regardless of programming's place.
What was also interesting was learning that some of the guys had already programmed in prison before. I heard a story of guys working for a client in Arizona. They ran into issues, however, when the client's programmer had a complex of working with programmers in prison. Sounds similar to folks having concerns with working with offshore developers, doesn't it? Probably the most interesting thing about this situation was that all communication about the code was done over telephone. No email, no instant message, no internet. Just telephone and books. That's tough!
After lunch I had the opportunity to see a 90 second space battle video that an inmate rendered using Blender and some other graphics tools. It was darn impressive. There was serious skill involved in making the video.
After the video we switched pairs, and I ended up with two fellows who I would classify as intermediate to advanced in programming skills. We were able to solve the problem through rotational ping pong pairing. Probably the most interesting thing about this was towards the end. We believed we were done, until we read the final step of the kata which told us that the program should ignore all points after a game is won. Writing tests for that feature drew up a bunch of tiny hidden bugs which gave me the opportunity to talk about self documenting code and extracting methods to handle those problems. Which, one of my triple knew quite well...probably better than I did, but was a bit too modest.
One of the things that occurred to me after the second pairing was how skilled some of these programmers were. I can honestly say that some of these guys had more skill than folks who I've been involved in interviews with on the outside. I believe a lot of it has to do with rote memorization. They have no Internet. Only books, Eclipse and each other. Any plugins and tools have to be brought in from the outside. They don't have StackOverflow to ask questions. Just each other, books, and whatever hints the IDE provides. So unlike most developers, who have adapted lazy instantiation of skills in order to adapt quickly to different technologies, these guys have had to battle harden the skills that they have. The other thing that surprised me, is how well they pair. One thing you might not expect in a prison situation is the ability to share information, reduce your ego, and pass the keys back and forth. Yet, they did it without concern. Of the advanced programmers, I had the luxury of pairing with, I saw folks who would have no problem fitting in with an Agile team. I know they could pass an audition, if they could get one on the outside.
So when I reflect on my experience, it makes me take a step back, and think about my own ego when I pair. Am I giving up the keyboard enough? Am I ping ponging enough? Am I letting a junior developer work through their problems enough, or do they need to shoot themselves in the foot a few times first? It was great reflection for me in handling my professional life.
One final note. The guys in here have made mistakes. Some of these guys have made really big mistakes. I didn't ask, but one of the inmates I paired with offered that he had been in the system for almost two decades. You can draw your own conclusions from that information. He had serious questions on if learning Java could help him when he got to the outside. I think it can, but as Dan put it at our retrospective dinner, the most important thing to learn is pairing and learning to work well with others. I believe the folks in this program were trying to do whatever they can to build skills that can help them later, be it on the inside or outside.
The most interesting thing about this experience, is how well it paralleled with the homilies given in my youth. It became obvious to me the older I got, that the priest was trying to say that, these prisoners know more about faith that you understand due to their situation. Similarly, these inmates knew more about programming than many folks I've encountered simply due to their faith that it's a skill that can help them later. They practiced hard, and were better than they would ever care to admit. Faith is something that isn't taught, but arrives, and it was inspiring to see.
I strongly encourage any programmer somewhere near Marion (any by near I mean 2.5 hour drive...we had folks from Ann Arbor show up), to try to make an effort to make it to a Code Retreat there. If you aren't...maybe a prison near you has a programming program you can volunteer in. It will be worth your time.
Tuesday, December 28, 2010
Random Cucumber/Ruby Things I Learned
Some random stuff I learned as a "seasoned" Java developer, working in Ruby and more specifically Cucumber for the first time this year, and not having a clue as to what I was doing. I realize some of this stuff will be mind numbingly obvious to "seasoned" Ruby devs, but thought it would be interesting to share regardless.
Some background. On my project some "legacy" Cucumber tests existed. Legacy, you ask? How can Cucumber tests be legacy? Simple, they were overly fragile, and violated DRY over and over again. Therefore, since my overall group has chosen Cucumber as the ATDD tool of choice (and by extension Gherkin as the Story card language of choice), it was a good idea to get in and refactor the tests. I had a good example in how to do this in the blog posts by Jeff Morgan, and I fortunately had someone who I could tap on the shoulder on a sister team in Joel Helbling. So with the goal of learning some Ruby in mind, and a copy of RubyMine I began tearing up the soil and planting cukes.
This works out pretty well, just don't check in your Gemfile.lock. Ideally your QA testers will be running the Watir versions, while CI will maintain sanity when they can't. If you don't have QA testers, then I suggest having two versions of the project open, with separate Gemfile.lock that represent each platform.
Firstly, when you're working with multiple versions of Ruby, in my case, JRuby and Ruby for Windows, you might need to modify your Gemfile by the platform. In my case, I needed JRuby to run the tests the same way as the legacy tests were running. JRuby drove Celerity (yay HtmlUnit!) tests on Hudson, while native Ruby drives Waitr/FireWatir browser driven versions of the same tests.
So, although it's very obvious after the fact, here's a clue...there is no Celerity for Ruby. HtmlUnit is a Java library, and Celerity is just a Waitr API based wrapper of it. So it only works on JRuby. I thought the Gemfile would just handle that, but I was very very wrong. Therefore, you have to do something like this:
platforms :jruby do
gem "celerity", "0.8.2"
gem "syntax"
end
if RUBY_PLATFORM =~ /mswin|mingw/ #Jruby no likey mingw
platforms :mswin, :mingw do
gem "watir", "1.6.7"
gem "watir-webdriver"
gem "win32console"
gem "win32-clipboard"
end
end
This works out pretty well, just don't check in your Gemfile.lock. Ideally your QA testers will be running the Watir versions, while CI will maintain sanity when they can't. If you don't have QA testers, then I suggest having two versions of the project open, with separate Gemfile.lock that represent each platform.
I'm certain there's a better way to do this...but I've not discovered it yet.
Another random thing. If you want to pass a parameter (like a table) from one Cucumber step definition to another, you use a comma at the end of the line and pass the parameter in.
In my case, I was developing tests that triggered various UI based errors on each page during a process to say retrieve a password.
So I had a step definition that said:
When /^I forget my password and enter the credentials$/ do |table|
# table is a |username|account_number|
#blah blah blah with username and account_number
end
But what do I do when I have passed this step and need to enter security questions? I want to go through the successful parts of this step to enter the next step. How do I do this?
When /^I forget my password and enter the credentials and answer the security questions$/ do |table|
# table is a |username|account_number|answer1|answer2|answer3|
When 'I forget my password and enter the credentials', table
# Do stuff to answer the security questions
See that comma at the end of the step definition call? It passes table to the next step and lets it grab username and account_number out of the hash. I couldn't find this anywhere in documentation, which means it's painfully obvious to Ruby folks, but it's not if you don't know it. I'm sure it's documented somewhere, but I've failed at finding it. But boy discovering that was quite useful.
One last thing that I can think of right now. For some reason the version of JRuby I have to use doesn't like to use rspec. So similarly, to the first problem, we need to do strange things to make JRuby happy in env.rb:
require 'rubygems'
if RUBY_PLATFORM =~ /java/ #JRuby is strange
require 'spec'
else
require 'rspec'
end
Hopefully these tips will be of some use to folks like myself who don't know enough Ruby to be dangerous...yet.
Wednesday, August 18, 2010
Cleaning up after maven-eclipse-plugin
There are a few of us out there who are in situations where we cannot use the m2eclipse plugin to run our Maven based projects without pain. To resolve that pain there's the maven-eclipse-plugin.
Unfortunately, the current version of the plugin has a bug which causes an in issue where duplicate facet entries are placed in the org.eclipse.wst.common.project.facet.core.xml file.
As an example:
Here's the Groovy script:
I'm sure there's some code golf you could play to shorten this up, but it does the trick. When you run it, the two extra entries in the file are gone, and a harmless backup file is left behind.
Unfortunately, the current version of the plugin has a bug which causes an in issue where duplicate facet entries are placed in the org.eclipse.wst.common.project.facet.core.xml file.
As an example:
How to solve this problem? A convient excuse to try out some Groovy and the GMaven plugin. Fortunately, Groovy is pretty good at modifying XML, and GMaven is pretty good at playing within the maven pom.
<faceted-project>
<fixed facet="jst.java"/>
<fixed facet="jst.utility"/>
<installed facet="jst.utility" version="1.0"/>
<installed facet="jst.java" version="5.0"/>
<installed facet="jst.java" version="5.0"/>
<installed facet="jst.utility" version="1.0"/>
</faceted-project>
Here's the Groovy script:
if (project.packaging != 'pom') {
def facetXml = new File(project.basedir, '.settings/org.eclipse.wst.common.project.facet.core.xml')
def facets = new XmlParser().parse(facetXml)
def facetSet = [] as Set
log.info('Loaded Eclipse Facet XML')
facets.installed.each {
if (! facetSet.add(it.'@facet')) {
it.parent().remove(it)
log.info('Removed ' + it.'@facet')
} else {
log.info('Put in ' + it.'@facet')
}
}
log.info('backup old XML')
new File(project.basedir, '.settings/org.eclipse.wst.common.project.facet.core.xml.bkp').delete()
facetXml.renameTo(new File(project.basedir, '.settings/org.eclipse.wst.common.project.facet.core.xml.bkp'))
log.info('writing items')
def newFacetXml = new File(project.basedir, '.settings/org.eclipse.wst.common.project.facet.core.xml')
new XmlNodePrinter().print(facets)
PrintWriter pw = new PrintWriter(newFacetXml)
xmlNodePrinter = new XmlNodePrinter(pw)
xmlNodePrinter.setPreserveWhitespace(true)
xmlNodePrinter.print(facets)
pw.close()
}
I'm sure there's some code golf you could play to shorten this up, but it does the trick. When you run it, the two extra entries in the file are gone, and a harmless backup file is left behind.
Wednesday, June 02, 2010
Sabotaging Your IT Team, Straight From 1944.
Today I ran across a highly entertaining post on Volokh which links to a manual on sabotage used during World War II. Most of the stuff in the manual is standard stuff, fires, blowing up machinery, etc. However, the really entertaining stuff begins on page 28, which is the "corporate" section. It details how to cause corporations to drive to a halt.
There are many other blogs that could dissect and twist this into any number of ways. Most of them have noticed how government tends to follow all of the recommendations as standard operation procedure. But since this is a blog which has primarily focused on programming, I'll spin this towards Agile/Lean project methodologies and the old Waterfall standby. And some of this I'll just spin towards IT in general.
So open up the PDF and skip ahead to page 28 of the Simple Sabotage Field Manual (page 32 in the PDF) and lets enjoy the mayhem. Some of these are a bit harder to connect, but they're spinnable, and what's the fun without spin?
Step a-1: Insist on using "channels". No changes to your project to deliver what is actually needed when you notice it. No team authority. Push your request up, get the change orders done, then work like hell to make it happen.
Step a-2: Make speeches. You know the guy who rolls into a project tells you you're doing it all wrong, and your management listens and tells you to do it a different way? That's this step.
Step a-3: Refer matters to committee. Same as a-1.
Step a-4: Bring up irrelevant issues. Attempting to change work in the middle of a development iteration/sprint. Unless you've got a major production outage, the issue can wait until the end of your work cycle.
Step a-5: Haggle over precise wording. This is more of a problem when you cannot get good copy up front. It can delay your work in waterfall and agile, so this one is just bad all around.
Step a-6: Reopen closed issues. Reopening the design of the system without a major problem that you actually need to solve. Creating more work for works sake, not because there's debt to be paid off.
Step a-7: Advocating "caution". Extra concern raised due to the unknown. This is bad for a team since it reduced the confidence to achieve success.
Step a-8: Be worried about authority to make a decision. Being afraid to make a decision since you might not have authority. You should have authority. This is similar to a-1, a-3.
Step b-1: Demand Written Orders. Well, we all need specs, but in Agile/Lean, we're getting those as we need them. We don't require the entire system be specified.
Step b-2: Misunderstand Orders. Quibbling over specs. This seems to occur when the business is not involved and rolls in several months down the line and says this isn't what they needed. Eliminate using Show and Tells.
Step b-3: Don't deliver until completely ready. Big Bang approach to delivery of software. Yep, that's Waterfall.
Step b-4: Don't order new working stocks until completely gone. This is what is known as Kanban isn't it? Pull work into the queues. There is no pull, only the schedule.
Step b-5: Order high quality materials which are hard to get. I'll change this one. Order expensive software/hardware that require expensive maintainence contracts. Especially when there's a well maintained open source project that your company could fund 2 developers to work on full time at a much lower cost.
Step b-6: Don't prioritize properly. When the business does not work with the development team to prioritize features, then the wrong thing gets prioritized by one of the two parties involved.
Step b-7: Insist on being perfect. Misuse of metrics. Insisting on 100% code coverage, zero issues in your quality checks as though it will eliminate Technical Debt. It won't. It will just make you move slower and ignores fixing the most expensive debt.
Step b-8: Mistakes in routing. Sorry I got nothing here.
Step b-9: Give trainees misleading instructions. This is more of a problem of new programmers not given enough time to gel into the new team. Train them through pair programming.
Step b-10: Be pleasant to ineffective workers, give them promotions. Um, Dilbert cornered the market on this one.
Step b-11: Hold conferences when there's important work to be done. Eliminate meetings. Prime Agile/Lean principle.
Step b-12: Multiply paper work. Unnecessary documentation for sure, but also poor technical design which doesn't allow for code reuse.
Step b-13: Multiply procedures. Too many people required to approve change controls, code sign off. Increased bureaucracy. Agile, Lean, ITIL, even Six Sigma applied to software have reduced these issues. And more than likely, unless their a micromanaging jerk, the upper management doesn't want to be that involved.
Step b-14: Apply all regulations to the last letter. Ever have a simple problem in a production install. Say a test property file was installed instead of production and the entire project is backed out rather than let the team fix the problem on the fly? Yeah, me too.
There's more comedy gold in here, but since I like to blog by the seat of my pants without editing, I need to stop. Heck, I've got a 1pm meeting to attend that is, well...let's just say that a-1, a-4, a-5 are to blame for this b-11.
There are many other blogs that could dissect and twist this into any number of ways. Most of them have noticed how government tends to follow all of the recommendations as standard operation procedure. But since this is a blog which has primarily focused on programming, I'll spin this towards Agile/Lean project methodologies and the old Waterfall standby. And some of this I'll just spin towards IT in general.
So open up the PDF and skip ahead to page 28 of the Simple Sabotage Field Manual (page 32 in the PDF) and lets enjoy the mayhem. Some of these are a bit harder to connect, but they're spinnable, and what's the fun without spin?
Step a-1: Insist on using "channels". No changes to your project to deliver what is actually needed when you notice it. No team authority. Push your request up, get the change orders done, then work like hell to make it happen.
Step a-2: Make speeches. You know the guy who rolls into a project tells you you're doing it all wrong, and your management listens and tells you to do it a different way? That's this step.
Step a-3: Refer matters to committee. Same as a-1.
Step a-4: Bring up irrelevant issues. Attempting to change work in the middle of a development iteration/sprint. Unless you've got a major production outage, the issue can wait until the end of your work cycle.
Step a-5: Haggle over precise wording. This is more of a problem when you cannot get good copy up front. It can delay your work in waterfall and agile, so this one is just bad all around.
Step a-6: Reopen closed issues. Reopening the design of the system without a major problem that you actually need to solve. Creating more work for works sake, not because there's debt to be paid off.
Step a-7: Advocating "caution". Extra concern raised due to the unknown. This is bad for a team since it reduced the confidence to achieve success.
Step a-8: Be worried about authority to make a decision. Being afraid to make a decision since you might not have authority. You should have authority. This is similar to a-1, a-3.
Step b-1: Demand Written Orders. Well, we all need specs, but in Agile/Lean, we're getting those as we need them. We don't require the entire system be specified.
Step b-2: Misunderstand Orders. Quibbling over specs. This seems to occur when the business is not involved and rolls in several months down the line and says this isn't what they needed. Eliminate using Show and Tells.
Step b-3: Don't deliver until completely ready. Big Bang approach to delivery of software. Yep, that's Waterfall.
Step b-4: Don't order new working stocks until completely gone. This is what is known as Kanban isn't it? Pull work into the queues. There is no pull, only the schedule.
Step b-5: Order high quality materials which are hard to get. I'll change this one. Order expensive software/hardware that require expensive maintainence contracts. Especially when there's a well maintained open source project that your company could fund 2 developers to work on full time at a much lower cost.
Step b-6: Don't prioritize properly. When the business does not work with the development team to prioritize features, then the wrong thing gets prioritized by one of the two parties involved.
Step b-7: Insist on being perfect. Misuse of metrics. Insisting on 100% code coverage, zero issues in your quality checks as though it will eliminate Technical Debt. It won't. It will just make you move slower and ignores fixing the most expensive debt.
Step b-8: Mistakes in routing. Sorry I got nothing here.
Step b-9: Give trainees misleading instructions. This is more of a problem of new programmers not given enough time to gel into the new team. Train them through pair programming.
Step b-10: Be pleasant to ineffective workers, give them promotions. Um, Dilbert cornered the market on this one.
Step b-11: Hold conferences when there's important work to be done. Eliminate meetings. Prime Agile/Lean principle.
Step b-12: Multiply paper work. Unnecessary documentation for sure, but also poor technical design which doesn't allow for code reuse.
Step b-13: Multiply procedures. Too many people required to approve change controls, code sign off. Increased bureaucracy. Agile, Lean, ITIL, even Six Sigma applied to software have reduced these issues. And more than likely, unless their a micromanaging jerk, the upper management doesn't want to be that involved.
Step b-14: Apply all regulations to the last letter. Ever have a simple problem in a production install. Say a test property file was installed instead of production and the entire project is backed out rather than let the team fix the problem on the fly? Yeah, me too.
There's more comedy gold in here, but since I like to blog by the seat of my pants without editing, I need to stop. Heck, I've got a 1pm meeting to attend that is, well...let's just say that a-1, a-4, a-5 are to blame for this b-11.
Monday, December 28, 2009
TrExMa 0.8.6
TrophyManager again changed their Squad page. I've finally been able to find some time around the holidays to fix the TrExMa plugin.
It's available at Google Code as version 0.8.6.
It's available at Google Code as version 0.8.6.
Tuesday, November 17, 2009
Simple Web Tool Building With Maven (The Woot-Off Tracker Revisited)
One of my favorite tools of the past year was the Woot-Off tracker I developed just over a year ago. In my mind, one of the problems this tool had was that I needed to start up a proxy server to host make calls on behalf of the AJAX Javascript. I needed to find some way to make this easier. Additionally, I had run into an issue when I actually tried to run the tool behind a firewall proxy.
Since discovering the power of Maven, I thought I'd look into seeing what was available for the Winstone servlet engine I used on the first iteration of this tool. Conviently, there was the winstone-maven-plugin, a mojo which, among other things, built an executable jar for winstone with your war already deployed.
This was perfect for my needs. So (along with converting to apache's http-client) I set out on a new maven adventure. My pom looked like this:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.twofourone</groupId>
<artifactId>woot.tracker.proxy</artifactId>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>
<name>woot.tracker.proxy</name>
<url>http://twofourone.blogspot.coom</url>
<dependencies>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet.jsp</groupId>
<artifactId>jsp-api</artifactId>
<version>2.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.6</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.0.2</version>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
<plugin>
<groupId>net.sf.alchim</groupId>
<artifactId>winstone-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>embed</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>net.sf.alchim</groupId>
<artifactId>winstone-maven-plugin</artifactId>
<version>1.2</version>
<configuration>
<cmdLineOptions>
<property>
<name>httpPort</name>
<value>8480</value>
</property>
<property>
<name>ajp13Port</name>
<value>-1</value>
</property>
<property>
<name>controlPort</name>
<value>-1</value>
</property>
<property>
<name>directoryListings</name>
<value>false</value>
</property>
<property>
<name>useInvoker</name>
<value>false</value>
</property>
</cmdLineOptions>
</configuration>
<executions>
<execution>
<goals>
<goal>embed</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
</plugins>
<finalName>woot.tracker.proxy</finalName>
</build>
</project>
We have two setups for winstone. The first goal, embed, tells Maven to stick winstone into the jar during the package phase. By default, this step will create a jar with the same name as your project and throw -standalone on the end. The second setup tells winstone what port it should run on, etc. These could potentially be in a single setup, just haven't tried it yet.
In order to run the proxy, you simply execute the jar from a command prompt and you'll have a war deployed to port 8480.
How has our proxy changed? In order to support a firewall call, I decided to use Apache's HttpClient. Adding the dependency for HttpClient to the pom automagically downloads all dependencies of HttpClient for use by the war. Our new code looks like this:
package com.twofourone.woot.tracker.proxy;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.impl.client.DefaultHttpClient;
/**
* Proxy Servlet to redirect requests to the Woot site.
* @author mcornell
*/
public class WootProxy extends HttpServlet {
static final long serialVersionUID = 1L;
private HttpClient httpClient = new DefaultHttpClient();
private HttpGet wootGet;
/**
* Sets up the HttpClient with proxy info if provided. Sets up the URL for Woot.
*/
@Override
public void init() throws ServletException {
String proxyHost = System.getProperty("proxyHost", "");
String proxyPort = System.getProperty("proxyPort", "");
if (proxyHost.length() > 0 && proxyPort.length() > 0) {
HttpHost proxy = new HttpHost(proxyHost, Integer.parseInt(proxyPort));
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
}
String contentObj = "http://www.woot.com/salerss.aspx";
wootGet = new HttpGet(contentObj);
}
/**
* Processes requests for both HTTPGETandPOSTmethods.
* @param request servlet request
* @param response servlet response
* @throws ServletException if a servlet-specific error occurs
* @throws IOException if an I/O error occurs
*/
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
HttpResponse wootResponse = httpClient.execute(wootGet);
HttpEntity wootEntity = wootResponse.getEntity();
response.setContentType(wootEntity.getContentType().toString());
// Get and read the input stream.
StringBuffer buffer = new StringBuffer();
BufferedReader din =
new BufferedReader(new InputStreamReader(wootEntity.getContent()));
String s;
while ((s = din.readLine()) != null) {
buffer.append(s);
}
din.close();
// Now write the bytes out to the client.
byte[] contentBytes = buffer.toString().getBytes();
OutputStream out = response.getOutputStream();
out.write(contentBytes, 0, contentBytes.length);
out.flush();
out.close();
}
/**
* Handles the HTTPGETmethod.
* @param request servlet request
* @param response servlet response
* @throws ServletException if a servlet-specific error occurs
* @throws IOException if an I/O error occurs
*/
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
processRequest(request, response);
}
/**
* Handles the HTTPPOSTmethod.
* @param request servlet request
* @param response servlet response
* @throws ServletException if a servlet-specific error occurs
* @throws IOException if an I/O error occurs
*/
@Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
processRequest(request, response);
}
/**
* Returns a short description of the servlet.
* @return a String containing servlet description
*/
@Override
public String getServletInfo() {
return "Woot Tracker Proxy";
}
}
The code has changed to setup the client once and initialize it once. Additionally, there is the code provided by the HttpClient project that adds a proxy to the client. One additional change needs to be the ability to track a Kids.Woot-Off URL in case it's a different kind of woot-off.
The HTML/Javascript remains the same.
Friday, October 30, 2009
Converting to the Ultimate Java Build Stack
Over the past 6 months, I've had the pleasure and pain of working with Maven. Even with all of the pain, I've found myself a fan of the technology, assuming, of course, that you have the appropriate Maven stack set up.
During my time at the client, I setup Hudson, Nexus, and Sonar as outlined in Christopher Judd's blog post. We had actually already decided upon Hudson and Nexus through discovery before Judd published his post. Sonar was a nice discovery we added to the mix a bit later. Having setup this stack, I thought I'd offer some thoughts on my experiences.
I cannot stress how important having Nexus (or Artifactory) is to keeping your Maven projects running smoothly and effectively. Having a repository management tool and proxy allows for all of your developers (and Hudsons) to simply point at the proxy and download everything required for a build quickly. It also provides a place for you to host your third party jars from apps that have been purchased by your organization. Not everything is open source of course.
The steps are simple enough, install Nexus and setup your settings.xml file to point to it. For developers, they can use their own settings.xml file. For Hudson, you may want to modify the settings.xml in your Maven install (especially if you're running on Windows). You might need to add a mirror for Java.net's Maven 2 repository. You might not. It depends upon how up to date Maven's central repository is.
Sonar was a nice surprise. Providing PMD, Checkstyle and Cobertura results in a nice visual manner that allowed for quick feedback was surprisingly effective. As a developer, I found it to be my favorite part of the switch to Maven. Sure, Maven provides this information as part of its site goal, however, it doesn't present it like this.
Obviously there are huge gains to be made by integrating Hudson and getting it to build upon version control commits, but I found more intriguing was its ability to trigger downstream builds. When I built the commons library for the app, every maven artifact that used the commons library was triggered for build. Numerous times, I discovered that I or someone else had broken something in a downstream library due to an API fix. That instant feedback (through RSS feeds) was an invaluable time saver.
What about Maven itself? What does it gain you? In the case I was working on, tremendous build granularity. The original build used Ant, and was built in a monolithic manner. Each component was built by the single script and bundled together. One could argue it wasn't the best ant script in the world, and they'd be right. Maven allowed us to break these dependencies apart. So now, instead of having one monolithic code base which included what were really five separate artifacts, we now had five separate projects, each self contained using dependencies to draw in the needed artifacts. As such the WAR and EAR builds ended up being very simple where they just drew in their needed dependencies and just worked.
During my time at the client, I setup Hudson, Nexus, and Sonar as outlined in Christopher Judd's blog post. We had actually already decided upon Hudson and Nexus through discovery before Judd published his post. Sonar was a nice discovery we added to the mix a bit later. Having setup this stack, I thought I'd offer some thoughts on my experiences.
I cannot stress how important having Nexus (or Artifactory) is to keeping your Maven projects running smoothly and effectively. Having a repository management tool and proxy allows for all of your developers (and Hudsons) to simply point at the proxy and download everything required for a build quickly. It also provides a place for you to host your third party jars from apps that have been purchased by your organization. Not everything is open source of course.
The steps are simple enough, install Nexus and setup your settings.xml file to point to it. For developers, they can use their own settings.xml file. For Hudson, you may want to modify the settings.xml in your Maven install (especially if you're running on Windows). You might need to add a mirror for Java.net's Maven 2 repository. You might not. It depends upon how up to date Maven's central repository is.
Sonar was a nice surprise. Providing PMD, Checkstyle and Cobertura results in a nice visual manner that allowed for quick feedback was surprisingly effective. As a developer, I found it to be my favorite part of the switch to Maven. Sure, Maven provides this information as part of its site goal, however, it doesn't present it like this.
Obviously there are huge gains to be made by integrating Hudson and getting it to build upon version control commits, but I found more intriguing was its ability to trigger downstream builds. When I built the commons library for the app, every maven artifact that used the commons library was triggered for build. Numerous times, I discovered that I or someone else had broken something in a downstream library due to an API fix. That instant feedback (through RSS feeds) was an invaluable time saver.
What about Maven itself? What does it gain you? In the case I was working on, tremendous build granularity. The original build used Ant, and was built in a monolithic manner. Each component was built by the single script and bundled together. One could argue it wasn't the best ant script in the world, and they'd be right. Maven allowed us to break these dependencies apart. So now, instead of having one monolithic code base which included what were really five separate artifacts, we now had five separate projects, each self contained using dependencies to draw in the needed artifacts. As such the WAR and EAR builds ended up being very simple where they just drew in their needed dependencies and just worked.
Additionally, Maven offered unit testing at a very cheap cost. The initial code base had no unit testing. The developers were keen to use the debugger to test scenarios, which is needed to understand the mass amounts of reflection which were in use. However, that's only helpful in understanding the code. There should have been unit tests which were stood up to test each component on its own. Maven automagically gives you unit testing with the surefire plugin enabled by default. Make sure JUnit is a test dependency and start writing tests into the test source tree. Keep your tests in the same package as your code and start gaining coverage. In between adding new features to the code base, I was able to increase code coverage on the base level components from 0 to 40%. Obviously there's still a long way to go, but this was a vast improvement!
One thing I will note about Maven is the tooling. There is tremendous tooling available in the m2eclipse plugin for Eclipse. Netbeans 6.5 and above opens Maven projects without needing to do much of anything. It just opens the directory containing the pom and imports the project. If you're moving to Maven, make sure that your development tools can support it. As an example, RAD 7.0.x is Eclipse 3.2. As such the m2eclipse plugin does not work for it. If you must use RAD 7.0.x, I would advise building your project outside of RAD. Use a more up to date IDE to get the projects started and structures built. Then use the maven-eclipse-plugin to build the proper project eclipse artifacts for your projects. It will still be a challenge to build proper EAR and WAR projects for RAD. Maven has a lot of help for this sort of pain. However it can only be applied if you use the proper tools.
I've got a bit more to say about Maven, especially with my experiences of converting projects to Maven, the various pain and joy points I experienced and overall satisfaction. But I'll leave that for another post. Until then, I strongly suggest that if you have new development and you are looking for a pure Java based project build management tool, that you should take a long hard look at Maven.
Monday, September 14, 2009
Wierdness with Maven's site-deploy and Hudson
So, for the past five months or so, I've been working on porting some Java projects to Maven and the Ultimate Enterprise Java Build System.
A quick aside, Sonar is a very underrated tool. You should try it. I find the reporting very easy to use and very handy for working through potential coding mistakes in the legacy code I've been working with.
One of those Java projects is a large multi-module Maven build. It has five different modules, three Jar modules, a War module and *gasp* an Ear module. It's been a bit of a beast to get going, but it seems to be working just fine.
On Monday, I began to perform releases to move from Maven SNAPSHOTs to actual real version numbers. Ah, the joys of SCM release management. Thankfully, all of this pain ends up being worth it to easily tag and lock the code down using the maven-release-plugin. There were a couple of bumps, but it wasn't bad.
The fun began when I built my Hudson jobs to perform release builds. These jobs simply pulled the tagged release and performed the following goals:
clean install site-deploy
Obviously, I used Hudson's deploy step for the artifacts rather than Maven's. And everything was fine, until I got this:
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Oops, what's that about? A little more background. This particular maven build was a multimodule build to create jar, war and ear files. The artifact it couldn't resolve was the first jar it built! Scanning the log file, it became clear that the job was not installing the built jar to the repository.
I don't know why that is, and am not certain if it's even worth opening a trouble ticket since the fix ended up being very simple. It seems that site-deploy was causing issues with the flow of the job within Hudson. By removing site-deploy the build ended up being successful. The jar was compiled, tested and installed before the next module's build was invoked.
I still wanted to build the site, so I investigated the M2 Extra Steps Plugin. This tool allows you to add multiple maven steps before and after the main job. I added the site-deploy goal to run after the main install goal and the sites were generated!
Monday, July 27, 2009
Trexma for Firefox 3.5
Been away from the blogging world for a while. Been away from the Trexma world for a while.
During that time, Firefox 3.5 was released and I hadn't had the opportunity to change a simple setting and redeploy the package for 3.5.
Well that's been done now. You can now download Trexma 0.8.5.1 from the google code site.
Hopefully this helps. :)
Tuesday, February 10, 2009
US Mexico Tomorrow (the Mexi-whore edition)
Going to go away from programming and into hijinks mode for a moment. Too many memories coming back from eight years ago during the epic 2001 battle between the US and Mexico in Columbus.
As with most big sporting events that actually allow in the inebriated public, the stands can be an interesting place. I stood in about 2 rows from the top of the North End, directly behind the goal. It was from this angle that I was able to see some mighty entertaining things.
The first was the Mexi-whore. Someone, obviously during a highly energetic brainstorming session fueled by Jager or Firewater, decided to purchase an adult doll. You know the kind of doll I'm talking about. The owner then acquired an undergarment for the doll, a thong of some nature, as well as a Mexican jersey. At various times during the match, the doll was held aloft by a leg while a cohort proceeded to spank the doll.
Needless to say, the presence of this doll has reached legendary status. But today I have found proof! In this snap from YouTube, you can see the head of the doll circled. It did exist. I can only hope it will make a return.
Other entertaining items were the signs. This sign would not made it through today's PC filters. The text of it read: Red Card, Yellow Card, Mexican
Yep, definitely wouldn't be "allowed" today. It also shows you how different things were eight years ago. Most everyone had a good time...some had too much of one. There was that one drunken guy who thought it would be a good idea to take taunting to the next level and to moon a bunch of Mexican fans in the parking lot. It might have been harmless had their not been a slew of vulgarity and elementary school kids in the car. Needless to say, he almost got his ass kicked.
The full YouTube video for your enjoyment:
At any rate, the hijinks tomorrow should be outstanding. If you're attending, I hope to see you there. If you're watching at home, hopefully you'll see some antics on TV or YouTube in the days to come. I'm hoping for a big US win, in the slop and wind that will be Columbus Crew Stadium. And maybe the return of the Mexi-whore.
As with most big sporting events that actually allow in the inebriated public, the stands can be an interesting place. I stood in about 2 rows from the top of the North End, directly behind the goal. It was from this angle that I was able to see some mighty entertaining things.
The first was the Mexi-whore. Someone, obviously during a highly energetic brainstorming session fueled by Jager or Firewater, decided to purchase an adult doll. You know the kind of doll I'm talking about. The owner then acquired an undergarment for the doll, a thong of some nature, as well as a Mexican jersey. At various times during the match, the doll was held aloft by a leg while a cohort proceeded to spank the doll.
Needless to say, the presence of this doll has reached legendary status. But today I have found proof! In this snap from YouTube, you can see the head of the doll circled. It did exist. I can only hope it will make a return.
Other entertaining items were the signs. This sign would not made it through today's PC filters. The text of it read: Red Card, Yellow Card, Mexican
Yep, definitely wouldn't be "allowed" today. It also shows you how different things were eight years ago. Most everyone had a good time...some had too much of one. There was that one drunken guy who thought it would be a good idea to take taunting to the next level and to moon a bunch of Mexican fans in the parking lot. It might have been harmless had their not been a slew of vulgarity and elementary school kids in the car. Needless to say, he almost got his ass kicked.
The full YouTube video for your enjoyment:
At any rate, the hijinks tomorrow should be outstanding. If you're attending, I hope to see you there. If you're watching at home, hopefully you'll see some antics on TV or YouTube in the days to come. I'm hoping for a big US win, in the slop and wind that will be Columbus Crew Stadium. And maybe the return of the Mexi-whore.
Friday, January 30, 2009
TrExMa 0.8.5 is out
TrExMa 0.8.5 is released. This includes a bunch of code cleanup, removal of the old quick lineup in favor of the complete Hungarian Algorithm method as well as converting the Shortlist, and Youth Development to use the new codebase.
If all goes well, this could be the final "preview" before deprecating 0.6.2.
If all goes well, this could be the final "preview" before deprecating 0.6.2.
Thursday, January 22, 2009
The Hungarian Algorithm in Javascript
For the past 7 months or so, I've been dabbling in XUL/Javascript as evidenced by the posts you can find on here about TrophyManager.com and the TrExMa plugin I created for the game.
Recently, I got to tackle an old school problem. Only I didn't know it was old school at the time when I started. I suppose that's because I've not been doing hardcore Computer Science type problems in quite some time.
Here's the problem: In TrophyManager, you field a soccer team of 11 players, 10 field players and a keeper. TrExMa calculates a skill for each player on your roster to try to give the user an idea of what the best possible lineup would be. Selecting the keeper is easy. Take the guy with the highest value. Selecting the field players, not so much.
Each player has 14 "visible" attributes. Early players of the game used the instruction manual to determine which attribute is most important for the position the player is in. For example, a Forward must be good at Finishing and Heading, etc. Additionally, each player has a favorite position. When they play "away" from their position, their skill drops. So a player may have a high finishing skill, but since they like to play left back, they're not going to be a very good striker and would have their skill reduced anywhere from 4-40% depending upon a hidden adaptability attribute.
Anyway, how does one figure out the best team? I can set the adaptablity factor from 1-10% and we can then caclulate the skill for every position on the field for every player on the team. Sounds good, right? But how to pick it?
I first looked at the original TrExMa algorithm within Excel. The original app had a drop down to determine the depth of the search. Once I looked at the code it was confirmed. TrExMa used a series of nested loops to build the lineup. If I'm not mistaken it could run from O(n^5) to O(n^10) depending upon the complexity you chose. Wow, no wonder it got very slow. I needed something else. I also realized that the only way this algorithm would guaruntee success would be to run it at O(n^10) setting. Using Javascript, this was simply unacceptable. I couldn't get away with something running for so long.
My first instinct was to use an algorithm similar to Fit First. In this case, find the highest value on the board, and place the player there. cross him out and cross out the position. Do it again until you've filled the positions. Supposedly this runs in O(n log n). Of course, 10 is pretty small for me, as I've only 10 boxes to fill. It's pretty good, but doesn't guaruntee the best match of players to positions. Why? Let's say you've got a player who is adept at a midfield and wing position. And another player who is only good at midfield. If the former is slightly better at midfield, he'll be placed there, and the other player will have to play somewhere else where he won't be as successful.
How do you correct this? I soon realized that this had to be a common problem, I just couldn't figure out what to call it? Eventually I discovered that it's known as an "assignment problem". It's an obvious name, now that I know what it is. The classical use of this problem is to determine the least overall cost of doing multiple things with multiple resources. It's very similar.
Soon I discovered the Hungarian Algorithm. It runs in O(n^4) (potentially O(n^3)). So it's not fast, but it's not insanely slow. It's not exponential, and it runs faster than the Excel app.
So what does the Hungarian Algorithm do? It uses a method of starring and priming to determine when a player is in its ideal place. Essentially, if two players are "best" at a position, it will not accept that assignment until it validates the best organization of those players. It's explained very well in the Wikipedia page.
There are many implementations of the Hungarian Algorithm on the net but none in Javascript. So here's one. I've pulled out some of the specifics for my implementation. You'll want to implement your own loadMatrix and getSolution methods. One thing to note, is that with my problem, we had to reverse the matrix. This is required because the Hungarian Algorithm uses small values to determine the best player (worker) for a position (job). I hope you find this useful.
Recently, I got to tackle an old school problem. Only I didn't know it was old school at the time when I started. I suppose that's because I've not been doing hardcore Computer Science type problems in quite some time.
Here's the problem: In TrophyManager, you field a soccer team of 11 players, 10 field players and a keeper. TrExMa calculates a skill for each player on your roster to try to give the user an idea of what the best possible lineup would be. Selecting the keeper is easy. Take the guy with the highest value. Selecting the field players, not so much.
Each player has 14 "visible" attributes. Early players of the game used the instruction manual to determine which attribute is most important for the position the player is in. For example, a Forward must be good at Finishing and Heading, etc. Additionally, each player has a favorite position. When they play "away" from their position, their skill drops. So a player may have a high finishing skill, but since they like to play left back, they're not going to be a very good striker and would have their skill reduced anywhere from 4-40% depending upon a hidden adaptability attribute.
Anyway, how does one figure out the best team? I can set the adaptablity factor from 1-10% and we can then caclulate the skill for every position on the field for every player on the team. Sounds good, right? But how to pick it?
I first looked at the original TrExMa algorithm within Excel. The original app had a drop down to determine the depth of the search. Once I looked at the code it was confirmed. TrExMa used a series of nested loops to build the lineup. If I'm not mistaken it could run from O(n^5) to O(n^10) depending upon the complexity you chose. Wow, no wonder it got very slow. I needed something else. I also realized that the only way this algorithm would guaruntee success would be to run it at O(n^10) setting. Using Javascript, this was simply unacceptable. I couldn't get away with something running for so long.
My first instinct was to use an algorithm similar to Fit First. In this case, find the highest value on the board, and place the player there. cross him out and cross out the position. Do it again until you've filled the positions. Supposedly this runs in O(n log n). Of course, 10 is pretty small for me, as I've only 10 boxes to fill. It's pretty good, but doesn't guaruntee the best match of players to positions. Why? Let's say you've got a player who is adept at a midfield and wing position. And another player who is only good at midfield. If the former is slightly better at midfield, he'll be placed there, and the other player will have to play somewhere else where he won't be as successful.
How do you correct this? I soon realized that this had to be a common problem, I just couldn't figure out what to call it? Eventually I discovered that it's known as an "assignment problem". It's an obvious name, now that I know what it is. The classical use of this problem is to determine the least overall cost of doing multiple things with multiple resources. It's very similar.
Soon I discovered the Hungarian Algorithm. It runs in O(n^4) (potentially O(n^3)). So it's not fast, but it's not insanely slow. It's not exponential, and it runs faster than the Excel app.
So what does the Hungarian Algorithm do? It uses a method of starring and priming to determine when a player is in its ideal place. Essentially, if two players are "best" at a position, it will not accept that assignment until it validates the best organization of those players. It's explained very well in the Wikipedia page.
There are many implementations of the Hungarian Algorithm on the net but none in Javascript. So here's one. I've pulled out some of the specifics for my implementation. You'll want to implement your own loadMatrix and getSolution methods. One thing to note, is that with my problem, we had to reverse the matrix. This is required because the Hungarian Algorithm uses small values to determine the best player (worker) for a position (job). I hope you find this useful.
/* Implementation of the Hungarian Algorithm to determine
* "best" squad. This is a "reverse" implementation.
* References:
* http://en.wikipedia.org/wiki/Hungarian_algorithm
* http://www.ams.jhu.edu/~castello/362/Handouts/hungarian.pdf (Example #2)
* http://www.public.iastate.edu/~ddoty/HungarianAlgorithm.html // Non-square
*/
var Hungarian = window.Hungarian = window.HG = {
/* 2 dimension arrays */
skillMatrix: null,
matrix: null,
stars: null,
/* Single arrays */
rCov: [],
cCov: [],
rows: 0,
cols: 0,
dim: 0,
solutions: 0, // "k"
FORBIDDEN_VALUE: -999999,
/* Rows MUST BE the Formation (Jobs)
* Columns MUST BE the Squad (Workers)
* Therefore, the Rows MUST BE PADDED
*/
hungarianAlgortithm: function(formation, squad) {
HG.init(formation, squad);
// Step 1
HG.matrix = HG.subtractRowMins(HG.matrix);
// Step 2
HG.findZeros(HG.matrix);
var done = false;
while (!done) {
// Step 3
var covCols = HG.coverColumns(HG.matrix);
if (covCols > HG.solutions -1) {
done = true;
}
if (!done) {
// Step 4 (calls Step 5)
done = HG.coverZeros(HG.matrix);
while (done) {
// Step 6
var smallest = HG.findSmallestUncoveredVal(HG.matrix);
HG.matrix = HG.uncoverSmallest(smallest, HG.matrix);
done = HG.coverZeros(HG.matrix);
}
}
}
return HG.getSolution(formation, squad)
},
init: function(formation, squad) {
HG.cols = squad.players.length;
HG.rows = formation.length;
HG.dim = Math.max(HG.rows, HG.cols);
HG.solutions = HG.dim;
HG.skillMatrix = HG.initMatrix(HG.rows, HG.cols);
HG.matrix = HG.initMatrix(HG.dim, HG.dim);
HG.stars = HG.initMatrix(HG.dim, HG.dim);
HG.matrix = HG.loadMatrix(squad, formation, HG.matrix, true);
HG.skillMatrix = HG.loadMatrix(squad, formation, HG.skillMatrix, false);
HG.rCov = new Array(HG.dim);
HG.cCov = new Array(HG.dim);
HG.initArray(HG.cCov, 0); // Zero it out
HG.initArray(HG.rCov, 0);
},
initMatrix: function(sizeX, sizeY) {
var matrix = new Array(sizeX);
for (var i=0; i<sizeX; i++) {
matrix[i] = new Array(sizeY);
HG.initArray(matrix[i], 0);
}
return matrix;
},
// Takes an array of positions as a formation.
// Takes a squad which contains an array of players
loadMatrix: function(squad, formation, matrix, reverse) {
matrix = loadYourMatrix(squad, formation, matrix); // I've removed my implementation here. Far too much stuff
if (reverse) {
// This reverses the matrix. We need to to create a cost based solution.
matrix = HG.reverseMatrix(HG.findMaxValue(matrix), matrix);
}
return matrix;
},
findMaxValue: function(matrix) {
var max = 0.0;
for (var i = 0; i < matrix.length; i ++) {
for (var j = 0; j < matrix[i].length; j++) {
if (matrix[i][j] > max) {
max = matrix[i][j];
}
}
}
return Number(max);
},
reverseMatrix: function(max, matrix) {
for (var i = 0; i < matrix.length; i ++) {
for (var j = 0; j < matrix[i].length; j++) {
matrix[i][j] = (Number(max) - Number(matrix[i][j])).toFixed(0);
}
}
return matrix;
},
subtractRowMins: function(matrix) {
for (var i = 0; i < matrix.length; i ++) {
var min = Number.MAX_VALUE;
for (var j = 0; j < matrix[i].length; j++) {
if (matrix[i][j] < min) {
min = Number(matrix[i][j]);
}
}
for (var k = 0; k < matrix[i].length; k++) {
matrix[i][k] = matrix[i][k] - min;
}
}
return matrix;
},
subtractColMins: function(matrix) {
for (var j = 0; j < matrix[0].length; j ++) {
var min = Number.MAX_VALUE;
for (var i = 0; i < matrix.length; i++) {
if (matrix[i][j] < min) {
min = Number(matrix[i][j]);
}
}
for (var k = 0; k < matrix[0].length; k++) {
matrix[k][j] = matrix[k][j] - min;
}
}
return matrix;
},
findZeros: function(matrix) {
for (var i = 0; i < matrix.length; i++) {
for (var j = 0; j < matrix[i].length; j++) {
if (matrix[i][j] == 0) {
if (HG.rCov[i] == 0 && HG.cCov[j] == 0) {
HG.stars[i][j] = 1;
HG.cCov[j] = 1;
HG.rCov[i] = 1;
}
}
}
}
// Clear Covers
HG.initArray(HG.cCov,0);
HG.initArray(HG.rCov,0);
},
initArray: function(theArray, initVal) {
for (var i = 0; i < theArray.length; i++) {
theArray[i] = Number(initVal);
}
},
coverColumns: function(matrix) {
var count = 0;
for (var i=0; i < matrix.length; i++) {
for (var j=0; j < matrix[i].length; j++) {
if (HG.stars[i][j] == 1) {
HG.cCov[j] = 1;
}
}
}
for (var k=0; k < HG.cCov.length; k++) {
count = Number(HG.cCov[k]) + Number(count);
}
return count;
},
/**
* step 4
* Cover all the uncovered zeros one by one until no more
* cover the row and uncover the column
*/
coverZeros: function(matrix) {
var retVal = true;
var zero = HG.findUncoveredZero(matrix); // Returns a Coords object..
while (zero.row > -1 && retVal == true) {
HG.stars[zero.row][zero.col] = 2 //Prime it
var starCol = HG.foundStarInRow(zero.row, matrix);
if (starCol > -1) {
HG.rCov[zero.row] = 1;
HG.cCov[starCol] = 0;
} else {
HG.starZeroInRow(zero); // Step 5
retVal = false;
}
if (retVal == true) {
zero = HG.findUncoveredZero(matrix);
}
}
return retVal;
},
findUncoveredZero: function(matrix) {
var coords = new HgCoords();
for (var i=0; i< matrix.length; i++) {
for (var j=0; j < matrix[i].length; j++) {
if (matrix[i][j] == 0 && HG.rCov[i] == 0 && HG.cCov[j] == 0) {
coords.row = i;
coords.col = j;
j = matrix[i].length;
i = matrix.length - 1;
}
}
}
return coords;
},
foundStarInRow: function(zeroRow, matrix) {
var retVal = -1;
for (var j = 0; j < matrix[zeroRow].length; j++) {
if (HG.stars[zeroRow][j] == 1) {
retVal = j;
j = matrix[zeroRow].length;
}
}
return retVal;
},
/**
* step 5
* augmenting path algorithm
* go back to step 3
*/
starZeroInRow: function(zero) { // Takes a Coords Object
TrU.log("Step 5: Uncovered Zero:" + zero.row + "," + zero.col, TrU.DEBUG );
var done = false;
var count = 0;
var path = HG.initMatrix(HG.dim*2, 2);
path[count][0] = zero.row;
path[count][1] = zero.col;
while (!done) {
var row = HG.findStarInCol(path[count][1]);
if (row > -1) {
count++;
path[count][0] = row;
path[count][1] = path[count - 1][1];
} else {
done = true;
}
if (!done) {
var col = HG.findPrimeInRow(path[count][0]);
count++;
path[count][0] = path[count - 1][0];
path[count][1] = col;
}
}
HG.convertPath(path, count);
// Clear Covers
HG.initArray(HG.cCov,0);
HG.initArray(HG.rCov,0);
HG.erasePrimes();
},
findStarInCol: function(col) {
var retVal = -1;
for (var i = 0; i < HG.stars.length; i++) {
if (HG.stars[i][col] == 1) {
retVal = i;
i = HG.stars.length;
}
}
return retVal;
},
findPrimeInRow: function(row) {
var retVal = -1;
for (var j=0; j< HG.stars[row].length; j++) {
if (HG.stars[row][j] == 2) {
retVal = j;
j = HG.stars[row].length;
}
}
return retVal;
},
/* Should convert all primes to stars and reset all stars.
* Count is needed to be sure we look at all items in the path
*/
convertPath: function(path, count) {
HG.logMatrix(path, "Step 5: Converting Path. Count = " + count);
for (var i=0; i < count+1; i++) {
var x = path[i][0];
var y = path[i][1];
if (HG.stars[x][y] == 1) {
HG.stars[x][y] = 0;
} else if (HG.stars[x][y] == 2) {
HG.stars[x][y] = 1;
}
}
},
erasePrimes: function() {
for (var i=0; i<HG.stars.length; i++) {
for (var j=0; j < HG.stars[i].length; j++){
if (HG.stars[i][j] == 2) {
HG.stars[i][j] = 0;
}
}
}
},
findSmallestUncoveredVal: function(matrix) {
var min = Number.MAX_VALUE;
for (var i = 0; i < matrix.length; i++) {
for (var j = 0; j < matrix[i].length; j++) {
if (HG.rCov[i] == 0 && HG.cCov[j] == 0) {
if (min > matrix[i][j]) {
min = matrix[i][j];
}
}
}
}
return min;
},
/**
* step 6
* modify the matrix
* if the row is covered, add the smallest value
* if the column is not covered, subtract the smallest value
*/
uncoverSmallest: function(smallest, matrix) {
TrU.log("Uncover Smallest: "+ smallest);
HG.logMatrix(matrix, "B4 Smallest uncovered");
for (var i = 0; i < matrix.length; i++) {
for (var j = 0; j < matrix[i].length; j++) {
if (HG.rCov[i] == 1) {
matrix[i][j] += smallest;
}
if (HG.cCov[j] == 0) {
matrix[i][j] -= smallest;
}
}
}
HG.logMatrix(matrix, "Smallest uncovered");
return matrix;
},
getSolution: function(formation, squad) {
var total = 0;
var lineup = [];
// Changed from length of stars, since we must ignore some rows due to padding.
for (var i = 0; i < HG.rows; i++) {
for (var j = 0; j < HG.cols; j++) {
if (HG.stars[i][j] == 1) {
/* the player (worker) at index j is the best player
* for poisition (job) at index i in your initial arrays.
*/
lineup.push(getThePlayer(i,j));
}
}
}
return lineup;
}
}
function HgCoords() {
this.row = -1;
this.col = -1;
}
Tuesday, January 06, 2009
New TrExMa Preview Release
I've updated TrExMa for Firefox after finding a bug in scraping the Transfer Pages.
This bug exists in all versions of the plugin as the Transfer List uses slightly different HTML to display starred skills. The starred skills were getting skipped over and treated as zero. Sheffield FC also picked up on this bug and reported to me as I was in the midst of adding some additional features for this release. So thanks to him for noticing the problem!
Additional features include the introduction of skill summaries, Attack, Defending, etc.
You can find the new plugin: http://code.google.com/p/trexma-for-firefox/
Thanks!
This bug exists in all versions of the plugin as the Transfer List uses slightly different HTML to display starred skills. The starred skills were getting skipped over and treated as zero. Sheffield FC also picked up on this bug and reported to me as I was in the midst of adding some additional features for this release. So thanks to him for noticing the problem!
Additional features include the introduction of skill summaries, Attack, Defending, etc.
You can find the new plugin: http://code.google.com/p/trexma-for-firefox/
Thanks!
Wednesday, December 17, 2008
TrExMa for Firefox 0.8 Preview Release
Over at TrExMa For Firefox, you can find a long awaited update to the TrExMa plugin for the popular TrophyManager game. Version 0.8 is available and this is a Preview Release.
This was an excuse to learn a little about XUL, do some jQuery and refactor the plugin. Unfortunately, it still needs a ton of refactoring, and the code is quite soupy. This is a use at your own risk version. The plugin can crash, provide a little popup saying something's wrong, and sometimes not even offer that. It's buggy. Since I'm not a XUL or UI expert, there's a lot of things that have been done in a sloppy fashion and things that cause bugs. Anyone who's using this, I would really appreciate using the issue tracker rather than sending an email over at TM or posting in those forums. Reason is, I get notified if someone posts an issue.
It allows the user to see hit players abilities for all positions and dynamically change the loss of skill for the player being out of a favorite position. This feature works by clicking on the TrExMa for Firefox label in the status bar. You'll see a little XUL window appear in your browser. If you browse to a squad screen or a transfer list screen, you'll get a list of players to choose from. Clicking on a player will present the skills for that player in all positions available.
In addition, a drop down box is available to determine what player is the best at each position. If you want to find the best ML, select ML and the plugin will produce the top 5 players on that screen in that particular position.
Quite a bit of refactoring involved brining in the jQuery Javascript plugin. I'm very happy with the integration of jQuery, it's an outstanding tool, as it just flat out works against HTML and XUL.
If you don't like it, uninstall it and reinstall 0.6.2. If you do like it, please offer suggestions and features that you'd like to see. Still on the idea block is the ability to determine your best 11 for a given formation, but that's a little ways off.
This was an excuse to learn a little about XUL, do some jQuery and refactor the plugin. Unfortunately, it still needs a ton of refactoring, and the code is quite soupy. This is a use at your own risk version. The plugin can crash, provide a little popup saying something's wrong, and sometimes not even offer that. It's buggy. Since I'm not a XUL or UI expert, there's a lot of things that have been done in a sloppy fashion and things that cause bugs. Anyone who's using this, I would really appreciate using the issue tracker rather than sending an email over at TM or posting in those forums. Reason is, I get notified if someone posts an issue.
It allows the user to see hit players abilities for all positions and dynamically change the loss of skill for the player being out of a favorite position. This feature works by clicking on the TrExMa for Firefox label in the status bar. You'll see a little XUL window appear in your browser. If you browse to a squad screen or a transfer list screen, you'll get a list of players to choose from. Clicking on a player will present the skills for that player in all positions available.
In addition, a drop down box is available to determine what player is the best at each position. If you want to find the best ML, select ML and the plugin will produce the top 5 players on that screen in that particular position.
Quite a bit of refactoring involved brining in the jQuery Javascript plugin. I'm very happy with the integration of jQuery, it's an outstanding tool, as it just flat out works against HTML and XUL.
If you don't like it, uninstall it and reinstall 0.6.2. If you do like it, please offer suggestions and features that you'd like to see. Still on the idea block is the ability to determine your best 11 for a given formation, but that's a little ways off.
Tuesday, December 09, 2008
A Spry Woot-Off Tracker
Although Flex is a neat tool, good ol' Javascript has been around for quite a while, and I thought I'd try to build a Woot-Off tracker using Flex's Javascript cousin, Spry. Conveniently, there's a Woot-Off going on today.
Spry is an interesting Javascript toolkit since it focuses on data extraction and presentation and widgets. In this example, we're not using any widgets, but we are taking advantage of the Spry Dataset tools to grok the XML stream from Woot. Just like with the AIR version, we need to use a proxy to grab the Woot XML stream.
Spry's DataSet works by allowing the developer to query the data set. Since Woot is providing an RSSish feed, we use the XMLDataSet class. By using braces, the dataset's contents can be accessed using the name of the XML tag. So to grab the price, we use {woot:price}. It's relatively simple.
The challenging part is for data that's not quite perfectly formatted. In this case, the percentage needs to be multiplied by 100. We do that inside the Observer. The Observer can update the contents of the dataset. So you can simply create an observer function and change away. The description also needs to be cleaned up since its HTML entities do not provide the needed effect. We actually want to use the tags, so we unentify them.
The timer is actually easier than in Flex, since the timer comes automatically with the DataSet with the loadInterval option. The only thing we need to do is speed it up and slow it down at the appropriate time.
And here's the CSS:
This application can be run inside Tomcat or wherever you have a proxy running.
Spry is an interesting Javascript toolkit since it focuses on data extraction and presentation and widgets. In this example, we're not using any widgets, but we are taking advantage of the Spry Dataset tools to grok the XML stream from Woot. Just like with the AIR version, we need to use a proxy to grab the Woot XML stream.
Spry's DataSet works by allowing the developer to query the data set. Since Woot is providing an RSSish feed, we use the XMLDataSet class. By using braces, the dataset's contents can be accessed using the name of the XML tag. So to grab the price, we use {woot:price}. It's relatively simple.
The challenging part is for data that's not quite perfectly formatted. In this case, the percentage needs to be multiplied by 100. We do that inside the Observer. The Observer can update the contents of the dataset. So you can simply create an observer function and change away. The description also needs to be cleaned up since its HTML entities do not provide the needed effect. We actually want to use the tags, so we unentify them.
The timer is actually easier than in Flex, since the timer comes automatically with the DataSet with the loadInterval option. The only thing we need to do is speed it up and slow it down at the appropriate time.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:spry="http://ns.adobe.com/spry">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />
<title>Spry Woot Tracker</title>
<link type="text/css" rel="stylesheet" href="http://yui.yahooapis.com/2.6.0/build/reset-fonts-grids/reset-fonts-grids.css"/>
<link href="css/wootTracker.css" rel="stylesheet" type="text/css" />
<script type="text/javascript" src="includes/xpath.js"></script>
<script type="text/javascript" src="includes/SpryData.js"></script>
<script type="text/javascript">
// we don't want to use cached data, and we need to reload every 30 seconds
var dsWootInfo = new Spry.Data.XMLDataSet("WootProxy", "rss/channel/item", {useCache: false, loadInterval: 30000});
var quickCheck = false; // Is the timer sped up
// Observer watches for when data changes and modifies for presentation
function wootObserver(notificationType, dataSet) {
if (notificationType == "onDataChanged") {
if (dataSet) {
var data = dataSet.getData();
var soldout = data[0]["woot:soldoutpercentage"];
data[0]["woot:soldoutpercentage"] = soldout*100;
var desc = data[0]["description"];
// Description contains HTML entities, fix them
// Something strange is going on with the formatter it's just desc not descdesc
desc = desc.replace(/>/g, ">");
desc = desc.replace(/</g, "<");
desc = desc.replace(/"/g, '"');
desc = desc.replace(/[\u201C\u201d]/g, '"');
data[0]["description"] = desc ;
// Determine if it's time to speed up or slow down
if (quickCheck) {
if (soldout < .95) {
dsWootInfo.stopLoadInterval();
dsWootInfo.startLoadInterval(30000);
quickCheck = false;
}
} else {
if (soldout > .95) {
dsWootInfo.stopLoadInterval();
dsWootInfo.startLoadInterval(1000);
quickCheck = true;
}
}
}
}
};
dsWootInfo.addObserver(wootObserver);
</script>
</head>
<body id="wootTracker">
<noscript>
<h1>This page requires JavaScript. Please enable JavaScript in your browser and reload this page.</h1>
</noscript>
<div id="doc" spry:region="dsWootInfo">
<h1 id="wootName"> {title} </h1>
<div style="float:left; padding-right: 10px">
<img src="{woot:thumbnailimage}" />
<h3 id="wootPrice">{woot:price} </h3>
<h4><a href="{woot:purchaseurl}" target="_blank">Buy This Woot</a></h4>
<h3 id="wootPercent">{woot:soldoutpercentage}% Sold Thus Far</h3>
</div>
<div id="description" >{description}</div>
</div>
</body>
</html>
And here's the CSS:
body {
background:#EDEDED none repeat scroll 0 0;
color:#191919;
margin:0;
padding:0;
}
h1 { font-size: 182%; margin-bottom: 0.5em}
h3 {font-size:138.5%; margin-bottom: 0.5em}
h4 {font-size:123.1%; margin-bottom: 0.5em}
li {list-style-type:disc; list-style-position:inside}
strong {font-weight:bold}
p {margin-top: 1em}
This application can be run inside Tomcat or wherever you have a proxy running.
Subscribe to:
Posts (Atom)