#27 - Towards a Ph.D. Thesis Proposal: NextGen Project and Optimization Strategy

With about 7 months remaining until a target graduation date, the first month has to entail projection and proposal.  The next three months from here til the end of year entail the core work and research on that proposal.  The final four months focus on developing the dissertation culminating that core work as well as attributing work done previously.  When I'm at a loss of ideas, I find that writing them out sometimes help them come forward.  It's like that in creative writing: don't think about what to write, just let the words come out.  Creativity and ideology live in the brain; writing can be an effective way for them to channel outward.

"What are my birds?" is asked by my advisor.  I have an algorithm called GALE, which can perform multi-objective optimization in very few evaluations.  This sounds really cool, but this is purely at this point, an algorithmic state of affairs.  Who cares?  What about the application of such an algorithm and its attributes to bettering the world?  This is the algorithms vs applications conflict.  Many theses and topics of research live purely in the world of algorithms, but lately a push for application-world research is called for.

GALE can optimize stuff, and learn stuff in few evaluations of the model.  If I want a thesis out of this, I need to find a way to tie the importance of this into the applications world.  Why would doing things faster be a good thing?  The only thing I can think of is for the affairs of safety-critical devices where speed is a requirement.  Secondly, what does it mean to learn stuff in few evaluations, blah, blah, really need to jump away from algorithmic-speak.

What The Birds Are

Overall, I have about four things in general that can be called my birds here.  First and foremost I have an algorithm called GALE.  This algorithm is a tool for software engineering researchers and intelligent systems design.  They key here, is that GALE can be used to aid in the development of systems that can analyze its environment and make expert decisions very quickly.  The obvious bit here is that it might be highly critical of the system to make those decisions quickly, e.g. consider systems where decisions affect the safety of human lives.  Furthermore, if the system cannot make those decisions quick enough, say, to react to very unexpected and sudden environment changes, then the safety of lives might also be endangered.

The the algorithmic world which GALE lives in, it needs a simulation to study as its environment.  In the application world, this simulation becomes the environment through machine learning.  This transition is a long way off, but the connection between optimization studies (like with GALE) and machine learning (using optimization to learn) is slowly becoming stronger every day.  It might be sensible to believe that one day in the future these two fields would merge and become one.

For now, we use GALE with a simulation in the algorithm world.  The first study on GALE was performed on a simulation called POM3, in which the process of completing a software project is modeled through requirements engineering (i.e. how best to plan a strategy of completing tasks to the project).  POM3 as a simulation has a handful of decisions that can be made, as well as a set of objectives to optimize in decision making.  Remember that anytime a decision is made, the critical-thinking process here is an aim to optimize some goal - e.g. what type of car should I buy?  We have goals of minimizing cost, maximizing MPG, maximizing aesthetic appeal, etc.)  For POM3, these goals were Completion (percentage of tasks that were completed, because a lot of projects only get so far before termination), Cost (total money spent), and Idle Rate (how often developers were waiting around on other teams).  The decisions were things like team size, size of the project, and other more domain-specific decisions to software engineering.

Going back to developing a thesis proposal; POM3 doesn't really fit our needs.  We'd like a simulation that can be optimized with GALE but has some application-world use where the power of making decisions quickly is very important.  POM3 isn't really safety-critical at all.  While it was a good and meaty simulation with many decisions and objectives, it just won't cut it for proposal in a thesis that wants to live in the application world.

The NASA Birds

My last two birds are two projects from my work out in California with NASA Ames Research Center.  These two projects involve aerospace research, and while one is a simulation, the other is a tool for ensuring safety of flight.  So it sounds like right away, we might have tools on hand for a thesis if we can combine the two projects.  WMC (Work models that compute) is the simulation, while TTSAFE (Terminal Tactical Safety Assurance Flight Evaluation) is the tool for conflict detection of aircraft - to make sure they don't collide in airspace.  Sounds trivial, but there's problems that need to be kept contained.

TTSAFE merely examines an input file containing codes that deal with aircraft locations, flight plans, tracking data, velocities, altitudes, and more.  This input file feeds into TTSAFE and a conflict detection algorithm determines if there are multiple aircraft heading towards each other on a collision course.  There are three main parameters here for such an algorithm.  The first is how often TTSAFE checks airspace for conflicts (granularity).  Secondly, to identify conflicts, a line can basically be drawn starting from every aircraft in airspace, and extending along the path of its velocity and direction.  If there are any two lines that intersect, then there is a conflict between the two aircraft of those two lines.  The second parameter here is how long to drawn the lines, e.g. 3 nautical miles, 10 nautical miles (or perhaps measured in time).  Thirdly, lines may not need to intersect to conflict, but instead merely come close to each other.  So the third parameter is the safe radius around the aircraft.

Once TTSAFE identifies conflicts, it proposes resolutions to those conflicts, and flight plans are adjusted based on rules of right-of-way in airspace.  There are problems with such a conflict detection algorithm because not all identified conflicts are truly a real conflict.  For instance, some aircraft may be on their way in making a turn, so while the velocity and current directions extend a line that intersect that of another aircraft, there would never have been a conflict because the aircraft was in the process of making a turn that "tricked" TTSAFE into believing there'd be a conflict.  Nevertheless, such a False Alarm is taken seriously and the aircraft is signaled to adjust flight plan - lengthening its miles traveled; costs more; takes longer to fly.  Overall, these are things we want to optimize, but false alarm rate is a major conflicting objective.

WMC is a project out of Georgia Tech which deals with computing trajectories for aircraft on approach to runways to land.  The computations rely on physics, aircraft type, and introduces cognitive measures that model the manner in which pilots take action in approaching the runway.  WMC is a simulation.  Taking only one aircraft at a time along with its flight plan and a starting point, starting velocity and starting altitude, it simulates the landing of that aircraft, yielding tracking data throughout to its completion.

After WMC simulates the landing of an aircraft, it can add its tracking data into TTSAFE along with the rest of airspace.  TTSAFE then determines if such a landing approach is safe, and if not, then resolutions are given and WMC re-computes the landing approach, and so on.  Since WMC only deals with landing, we need to realize that our "birds" here only deal with local airspace surrounding, say, 50 miles around an airport.  So there is no need to consider cross-country flights from takeoff to landing; but instead we would be in a sense, "spawning" aircraft randomly inside the 50 mile radius around an airport.  To further stress-test the system and emphasize the power of GALE, such extreme circumstances can be invented in airspace that otherwise might not naturally occur.  For example; what if an aircraft is hijacked and begins ignoring commands from ground control?  Or more calmly, what happens if some aircraft in general ignores a command and doesn't adjust flight plan?

How to Fly The Birds

TTSAFE and WMC on their own are fully developed.  The interaction between the two is not.  I'm wondering if it such a task could be completed timely in few short months.  TTSAFE is coded in Java, and WMC is coded in c++0x.  I can run them each on their own.  Furthermore, GALE is coded in Python.  Unless a clever bash script can tie everything together, I could be stuck figuring out how to fly the birds.  The main problem is figuring how to pass data between all three.  File I/O might be a bad idea, but the only probable one.  Going forward, I suppose the best thing is to take it one step at a time.  After all, four years ago I never thought I'd be able to be here because I looked at it as one giant step.  Instead, the many small steps are what got me here - and they are also the way forward.

Step 1) WMC has a problem with simulating only an aircraft of a predefined type.  It needs to be adjusted so that it simulates for an aircraft of an input type.  Thus, it should be able to read its parameter information from the BADA database for that aircraft type.

Step 2) Develop a script which can run WMC a bunch of times for random aircraft types, along with randomized parameters of decision nature (such as those for the cognitive models).

Step 3) Try to tie GALE with this WMC script from step 2.  If we can do this, then any roadblock with further connecting TTSAFE with everything should be understood and made simpler.  Then, although the practicality of optimizing WMC may not be understood, at the very least we can see how GALE runs with WMC (and how it runs using similar algorithms like GALE, i.e. NSGAII or SPEA2).    Note that I'm most worried about the "if we can do this" part.

Step 4) Make a script which generates an airspace of aircraft with variety of flight plans around a small radius about an airport.  This will be an input to the overall system that we ultimate aim to have.

Step 5) Feed the airspace of step 4 into WMC to generate accurate trajectory data for all aircraft.  This means we adjust the script of step 2, so that it instead doesn't generate random aircraft, but instead takes input from the airspace of step 4.  As for cognitive decision parameters, we use what we learned from step 3.

Step 6) We need a script now that feeds data from WMC back into TTSAFE.  Basically, we just need a way to adjust the tracking data in the airspace input file (made initially in step 4).

Step 7) Lastly, a script that combines all of these into a loop that runs until all aircraft land.  Then we compute statistics and metrics for the process - stuff we want to optimize.

Step 8) Step 4-7 become our ultimate model.  Whatever we call this, we then feed it through GALE vs NSGAII vs SPEA2 to optimize things and learn things, blah, blah.

Step 9) Adjustments, twitches, fixes.  Looking for and getting sane data from step 8.

Step 10) If we have sane data, then can we publish it?  i.e. can we go forward and put it all into a thesis proposal?

Krall Numbers

I'd always been interested in this kind of number I had discovered.  What you do is take any ordinary positive integer, most typical here is 7, and you look at all of its fractions under 1.  That is, look at the fractions 1/7, 2/7, ... all the way up to 6/7.  Expand those fractions, and then check out their decimal expansions.

Here, we see each respective decimal expansion in sequence.  The most interesting thing here is that there are patterns to be found in each expansion.  The pattern for the number 7 seems to be "142857".  This pattern cyclically repeats itself all throughout each expansion for the number 7.  The needed shift to match the cyclic pattern is also shown below, and the pattern being matched is indexed at the end of each line.

0.142857142857    , shift=  0,pattern#=  1,*
0.285714285714    , shift= -2,pattern#=  1,*
0.428571428571    , shift= -1,pattern#=  1,*
0.571428571429    , shift=  2,pattern#=  1,*
0.714285714286    , shift=  1,pattern#=  1,*
0.857142857143    , shift=  3,pattern#=  1,*

Some numbers have more than one pattern.  For example, the number 8 has 7 unique patterns.

0.125               , shift=  0,pattern#=  1,*
0.25                , shift=  0,pattern#=  2,**
0.375               , shift=  0,pattern#=  3,***
0.5                 , shift=  0,pattern#=  4,****
0.625               , shift=  0,pattern#=  5,*****
0.75                , shift=  0,pattern#=  6,******
0.875               , shift=  0,pattern#=  7,*******

However some numbers are a little more interesting.  There are identifiable patterns in the pattern index itself!  For example, check out number 11, where the stars at the end form a symmetric mountain of sorts.

0.09090909090909090909    , shift=  0,pattern#=  1,*
0.18181818181818181818    , shift=  0,pattern#=  2,**
0.27272727272727272727    , shift=  0,pattern#=  3,***
0.36363636363636363636    , shift=  0,pattern#=  4,****
0.45454545454545454545    , shift=  0,pattern#=  5,*****
0.54545454545454545455    , shift=  1,pattern#=  5,*****
0.63636363636363636364    , shift=  1,pattern#=  4,****
0.72727272727272727273    , shift=  1,pattern#=  3,***
0.81818181818181818182    , shift=  1,pattern#=  2,**
0.90909090909090909091    , shift=  1,pattern#=  1,*

And for number 13, we see again another symmetric pattern in the stars:

0.076923076923076923076923    , shift=  0,pattern#=  1,*
0.153846153846153846153846    , shift=  0,pattern#=  2,**
0.230769230769230769230769    , shift=  2,pattern#=  1,*
0.307692307692307692307692    , shift=  1,pattern#=  1,*
0.384615384615384615384615    , shift=  4,pattern#=  2,**
0.461538461538461538461538    , shift=  2,pattern#=  2,**
0.538461538461538461538462    , shift=  5,pattern#=  2,**
0.615384615384615384615385    , shift=  1,pattern#=  2,**
0.692307692307692307692308    , shift=  4,pattern#=  1,*
0.769230769230769230769231    , shift=  5,pattern#=  1,*
0.846153846153846153846154    , shift=  3,pattern#=  2,**
0.923076923076923076923077    , shift=  3,pattern#=  1,*

And perhaps, others have no pattern at all, such as for the number 37.  And others, have some truly odd quirks, such as in 26 when you look at the expansions for 22/26 and 23/26.  In fact, you see this a lot.

There's much we can learn and marvel at just by examining expansion sequences in this way.  You can try it yourself with this python script that I developed, located at: http://pastebin.com/a5UY0F0i.  To run the script, use "python krall_numbers.py 7" to run the test on the number 7.  Simply replace the number 7 with any number you want to experiment with.

#26 - The Art of Game Design - Chapter 30, 31, 32

These three chapters all focus on things a bit beyond the game.

Chapter 30 - The Game Transforms the Player

Jesse Schell brings up the topic of violence in video games.  I can't say I have any belief that the game transforms the player, but I do believe games change the way players think.  That is, games can be inspirational, but so can movies or books.  A lot of my core theories on fun center themselves around my big three - books, movies, games.  For me, they're all the same in some general theory on fun.  The second major topic in this chapter concerns the habit of addiction.  Agreeably, games are addictive.  Most entertainment simply is, however.

Chapter 31 - Designers have Certain Responsibilities

As a game designer, you find it in your interests as a hobby.  When you design games for the industry however, you are now representing the industry.  As a consequence, the industry also defines you.  Carrying that definition with you assigns you with responsibilities.  When you realize that your game can transform people, you realize that it is your duty to transform them positively.

Chapter 32 - Each Designer has a Motivation

Any game designer needs to understand their motivation.  It is the reason they get the job done.  But the question is, what exactly is your motivation?  And if it isn't worth your time, your motivation isn't strong enough.

#25 - The Art of Game Design Chapters 21,22,25

Chapter 21 - Some Games are Played with Other Players

This chapter introduces the concept of playing with other players, by stating that humans try to avoid being alone.  (Most humans at least.)  Multiplayer components are important to some games, because it provides humans a way to play the game and avoid being alone.  This chapter is largely just a precursor to the next, in which communities are the formed center of discussion in multiplayer games.

Chapter 22 - Other Players Sometimes Form Communities

The discussion of a community brings many topics to the table.  Players must have a way to take part in the community as though it were a real-life community.  For this, the lens of expression is introduced.  It is mentioned that at the heart of any community is conflict, but this is not always true - some games are cooperation based, and some are based purely on meeting others; i.e. a tea-party. moderators.

The lens of griefing is introduced to discuss the topic of players misbehaving in the game society.  Any community needs to be managed and policed; typically through game masters and

Lens #85: The Lens of Expression

Lens #86: The Lens of Community

Lens #87: The Lens of Griefing

Chapter 25 - Good Games Are Created Through Playtesting

Playtesting in this game is discussed as a component in game design that is necessary to ensuring and enhancing the fun of the game.  Although the author admits to hating to do playtesting; it is a crucial stage of design that he splits into four groups: focus groups, usability testing, playtesting (general testing) and QA testing.  Focus groups or surveys are sometimes powerful tools when done right.  Questions of interest to focus on are who (does the survey/testing), when (are you testing), what (are you testing), why, and where.

Lens #91: The Lens of Playtesting

#24 - The Art of Game Design Chapter 18,19,20

Chapter 18 - Worlds contain Characters

The breakdown of different character type adds some depth to my own understanding of a game, based on the three types of main media - books, movies and games.  But its important that a video game character be of any type  - they should all work.  For instance, the dimensions of character complexity can be a parameter to a game.  Another point to emphasize is that characters should be memorable, and one that adds to this is the complexity of the character name itself.  Names like Samus Aran come to mind - these type of names you don't see in the real world.  Because of their uniqueness, they tend to stick out and bring about an easier attachment to the overall experience (being or meeting) in the character.

An excellent example related to character trait is Chrono Cross.  The lead programmer for that game writes a script that "translates" the style of speak for the characters depending on which character it is that accompanies you throughout the main functions of the game.  The  story told by the character is the same regardless of which character you take.  But for example, you may take a pirate-type character or a french-dame with you, and the language is converted to reflect the type of character.

Lens #75: The Lens of the Avatar
Lens #76: The Lens of Character Function
Lens #77: The Lens of Character Traits
Lens #78: The Lens of the Interpersonal Circumplex
Lens #79: The Lens of the Character Web
Lens #80: The Lens of Status
Lens #81: The Lens of Character Transformation

Chapter 19 - Worlds Contain Spaces

The space of the world is that physical design world where the character can roam.  I would add to this, the manner in which the characters move in the world.  For instance, a lot of games emphasize continuous smooth movement, and some games focus on grid-based cell movement (like chess).

Lens #82: The Lens of Inner Contradiction
Lens #83: The Lens of The Nameless Quality

Chapter 20 - The Look and Feel of a World Is Defined by Its Aesthetics

This chapter seems like a recap of a lot of previous Lenses.  As a summary on aesthetics, it can help bring your game to life and add to the memorable experience.  It takes a real graphical eye to see that the aesthetics are proper - and it depends on the kind of game you build.  For example, a non-photorealistic approach may be more appropriate over a cell-shading cartoon style.  As another detail, sometimes people don't notice details.  The level of detail used may depend on the environment.  That seems funny to say.

#23 - JMOO Source Code Release

Search Based Software Engineering brings us a need to simulate real world problems and experiment with optimization of objectives.  For this, we need a way to model the simulations and the algorithms used to optimize objectives.  Formally, this is the field of Multi-Objective Optimization (MOO).  Sometimes, when there is only a single objective optimize, it is called Single-Objective Optimization instead.  In slightly older times, SOO was the more-studied, simplified case before its maturation and generalization into MOO.

MOO can best be described in mathematical terms.  MOO is used to optimize a Multi-Objective Problem (MOP).  A MOP consists of a set (X1, X2, …, Xn) of n decision explanatory variables, a set (Y1, Y2, …, Yk) of k objective response variables, and either a set of objective evaluation functions (f1(X1, X2, …, Xn), f2(X1, X2, …, Xn), …, fk(X1, X2, …, Xn)) or an evaluation model, either of which assigns values to each of the k objectives.   The decision variables each have their respective lower and upper bounds and in the case of constrained MOPs, the set of decisions as a whole is constrained to some set of bounds.  The objectives include an indication of which direction (minimizing, or maximizing) is optimal.  Typically, the optimization direction is ignored and an assumption is made, but I feel a need for its generalization.

As a broad overview, MOO is both: 1) an MOP and 2) an Algorithm to solve the MOP.  We just described the MOP above.  Now we discuss algorithmic approaches.  Most used is the MOEA (Multi-Objective Evolutionary Algorithm).  Two very state-of-the-art MOEAs are NSGA-II and SPEA2.  The MOEA is a standard type of algorithm that we build upon when presenting JMOEA (Joe's MOEA).

1. Initialization
2. Load Initial Population
3. Collect Initial Stats
4. Generational Evolution:
 - - 4a. Selector
 - - 4b. Adjustor
 - - 4c. Recombiner
 - - 4d. Collect New Stats
 - - 4e. Evaluate Stopping Criteria
5. Generational Representative Search

The outline above constitutes our JMOEA.  First, parameters are initialized, including MU, which defines the number of individuals in a population.  Each individual is a listing of decisions and objective fitness (when evaluated).  In the second step, an initial population is loaded - not generated.  This is so that when two algorithms try to optimize the same problem, they both begin with the same initial set of individuals.  Thirdly, stats are collected for the initial population.  The initial stats collection involves setting the reference point at the median of the population.  This median is then used to calculate our measures of quality and spread of the individuals in the population.  These measures help us determined how good we're doing in the algorithm.  Other stats include the medians of each objective and the number of evaluations accumulated thus far.

The core of our JMOEA is in step four when evolution starts.  Evolution is carried for a number of generations or until the stopping criteria of step 4e says to stop.  The methods of 4a, 4b and 4c are the steps that define the algorithm.  The selector first identifies a number of individuals from the population to use as "selectees", or offspring.  Then, the adjustor in step 4b modifies these selectees (usually per some chance rate).  Finally in 4c, the selectees and population are recombined, to either prune the size back down to MU-many individuals, or to grow back up to MU-many individuals.

The definition of an algorithm follows by assigning specific methods to each of 4a, 4b and 4c.  For instance, NSGA-II is defined when the selector is a tournament selection, adjustor is crossover and mutation, and the recombiner is the NSGA-II elite method (as described in its original paper reference).  SPEA2 is similarly defined, except its recombiner is defined to be the SPEA2 elitist method.

Finally, in step 5, we search through all the generations of the evolutionary process until we find a representing generation; ideally one of the better generations that we can use to say this is how good the algorithm worked.

The code that we use to develop JMOEA is contained in our JMOO package (Joe's MOO).  JMOO allows us to define decisions, objectives, problems, algorithms, individuals and the JMOEA.  It is very versatile in that new problems can very easily be defined, as well as new algorithms.  For fun: JMOO is our purple cow:

The code repository (python) for jmoo: http://unbox.org/things/tags/jmoo/

The rrsl method in jmoo_algorithms.py will require parts of http://unbox.org/things/tags/rrsl/

And the POM3 problem is at: http://unbox.org/things/tags/pom3/

#22 - POM3 Code Release

POM3 is the name given to my adaptation of POM2 (2009).  Similarly, POM2 is an adaptation of the original in 2008, POM.  Named for its originating authors, Port, Olkov and Menzies, POM was a simulation of the Requirements Engineering process in Software Engineering.

The job of the Requirements Engineer is to decide what gets implemented in a software project.  Typically this is done via priorities, but the prioritization strategies differ.  Traditionally, the Requirements Engineer deployed a Plan-Based strategy in which all requirements were prioritized once at the beginning of the project and never altered throughout the course of development.  In the alternative Agile development strategy however, requirements get re-prioritized during every iteration of development.  The question which POM tried to answer, is "which strategy is better?"

To answer the question, POM had to simulate many software projects very quickly.  And secondly, it had to evaluate those simulations.  Formally, this problem fits into the realm of Search Based Software Engineering. POM had become a Single-Objective Problem.  For the originating authors, the search for answering which strategy is better was a manual one involving the plot of parameters given 1,000 trials, and graphs.  In other words, a very inefficient approach at eyeing the results.  The authors of POM2 however, deploy a search technique in the algorithm known as KEYS.  In either case, the results are similar.  Plan-Based strategy is good in some cases, when requirements are stable, but Agile is better for when requirements are volatile and prone to change throughout development.

For more information, refer to references section.  [1] is the POM paper, and [2] is the POM2 paper.

POM3 however is no longer a Single-Objective Problem and is no longer a simulation intended only to focus on Requirements Engineering prioritization strategies, but rather a general software project simulation used to control our additional objectives in Cost, Idle and Completion (with Score).  Below, we discuss both the decisions and objectives of POM3 and then give an outline of the code, followed by some samples of the code.

POM3 Decisions

Decisions are those inputs to the POM3 simulation.  They are as follows.

1. Culture.  The percentage of personnel on the project that thrive on chaos.  Some projects are chaotic.  That is, a lot of requirements change and are unstable, i.e. volatile.
2. Criticality.  The impact to cost that safety-critical systems have upon projects.  Some projects have safety-critical requirements.  That is, some requirements are safety-critical in the event that failure absolutely cannot happen for the safety of human lives.
3. Criticality Modifier.  The percentage of teams that are dealing with safety-critical parts of the project.
4. Dynamism.  How dynamic is the project?  This affects the rate at which new requirements are found.
5. Interdependency.  A percentage of how many requirements depend on other requirements (assigned to other teams in the project).
6. Initial Known.  The percentage of the project that is initially known.  In the real world, we just don't know this, but we can guess.  In software world, all requirements exist, but some are marked as hidden.
7. Team Size.  The size of teams, in terms of how many personnel are on each team.
8. Size.  The project size.  Very small, small, medium, large, very large.
9. Plan.  The requirements prioritization strategy.  We provide 5 different schemes.

POM3 Objectives

Objectives are the outputs of the POM3 simulation that score how well the project was finished.

1. Score.  A score indicating the performance of the project's simulation.
2. Cost.  The average cost per completed requirement across all teams.
3. Idle.  The overall percentage of idleness of the project's teams.
4. Completion.  The overall percentage of how many visible requirements were completed.

POM3 Algorithm

The outline of the POM3 simulation is as follows.  We avoid core details while providing the code.

1. Initialization.
2. Requirements Generation.
3. Teams Generation.
4. Shuffling.
• For Each Team, Do: 
1. Assess Budget Information
2. Collect Available Tasks
3. Apply Sorting Strategy
4. Execute Available Tasks
5. Discover New Tasks
6. Update Task Priorities
5. Scoring.

This outline is contained in the pom3.py file.  The initialization step processes the inputs by containing them in a pom3_decisions class structure.  The number of shuffling iterations is determined.  And then we move on to step 2, where requirements are generated into a heap.  The code for this is handled in pom3_requirements.py, which further makes use of the pom3_requirements_tree.py file.  In step 3, we generate teams and assign parts of the requirements heap to each team.  The code that manages the assembly of teams is in pom3_teams.py, and the code that manages each team individually is in pom3_team.py.

Finally, shuffling occurs.  Only a few iterations of shuffling occurs, as determined back in the initialization step.  In each iteration, requirements are shuffled through each team over a series of steps.  At first, the team assesses their budget information and collects the money that is allocated to them based on their workload.  Secondly, the team collects any tasks that they can possibly work on.  Thirdly, they sort through these available tasks, according to some sorting strategy (requirements prioritization strategy).  Remember when I said Plan-Based strategies don't do this every iteration?  Well, in POM3 something happens during every iteration, so we're ignoring the traditional strategy for the sake of simplicity.  But it would be relatively simple to reintroduce it properly.  The reason we ignore it, is because quite simply, we don't care too much about it.

The fourth step of shuffling involves the execution of available tasks as possible, within budget limitations as assessed in step one of shuffling.  Fifthly, we look to discovering new tasks; which basically, all this means is marking some more requirements of the heap as visible which were previously invisible.  Finally, tasks are all updated on their 'values', i.e. priorities.  By the way, the code for each shuffling step is all contained in the pom3_team.py file, as all the work here is for each team individually.

Lastly, now that the simulation is finished, we score our objectives.  Score is the funkiest objective, as it considers each completed task's final priority value, as updated in every iteration as the optimal_value.  The true value is marked as well as they are completed in real time, before they are further updated via the last shuffling phase.  As a ratio, the score is calculated as frontier/optimal_frontier, where frontier = value/cost, and optimal_frontier = optimal_value / optimal_cost.  In the code, we refer to frontier as "our frontier", and the optimal frontier as the "god frontier", as it is information only a god-like entity could possibly know (we don't actually update the completed tasks in real life; but they get considered in the simulation for scoring purposes).

Extra

By the way, here's some fun python stuff.

class pom3_decisions:
    def __init__(p3d, X):
        p3d.culture = X[0]
        p3d.criticality = X[1]
        p3d.criticality_modifier = X[2]
        p3d.initial_known = X[3]
        p3d.interdependency = X[4]
        p3d.dynamism = X[5]
        p3d.size = int(X[6])
        p3d.plan = int(X[7])
        p3d.team_size = X[8]

The code above is a sample from POM3.  A simple class structure makes any instance of the pom3_decisions class an effective record of sorts.  When one instances the pom3_decisions class, they are making use of the class constructor (def __init__).  Now, the cool part is that unlike languages such as java, I can give my own name to the "this" reference.  In the sample above, I use the name p3d, and assign class data using it.  This is no different than using python's default "this" name, which is "self".  But sometimes, just sometimes - you can make the code look like syntactic candy with just the right name.

class pom3_requirements:
    def __init__(requirements, decisions):
        requirements.heap = requirements_tree()
        requirements.count = int(2.5*[3,10,30,100,300][decisions.size])
        requirements.decisions = decisions
        
        for i in range(requirements.count):
            

            ...

            
            parent = requirements.heap.tree[i]
            requirements.recursive_adder(parent, 1)

A heap is a bunch of trees crammed together.  Requirements.heap.tree[i] accesses the i'th tree of the heap.  Looks just like English.  But it's not, it's Python!

References

[1] D. Port, A. Olkov, and T. Menzies, “Using simulation to investigate requirements prioritization strategies,” in Automated Software Engineering, 2008. ASE 2008. 23rd IEEE/ACM International Conference on, Sept. 2008, pp. 268–277.

[2] Lemon, B.; et al., "Applications of Simulation and AI Search: Assessing the Relative Merits of Agile vs Traditional Software Development," in Automated Software Engineering, 2009. ASE 2009. 24th IEEE/ACM International Conference on, Nov. 2009, pp.580-584.