Monday, September 1, 2014

August Haskell Daily puzzles and solutions

August, 2014

  • August 1st, 2014: "But programming isn't math, so ..." Today's #haskell problem? 'Fix' yesterday's solution to be less ... 'icky.' π, fast (So you're not eating π? ... okay: 'quickly.') A solution set to today's exercises with Wow-o-wow results.
  • Bonus: August 1st, 2014: This is Embarrassing(ly parallel) to have π in da face!
  • A-to-the-ST for da D-down-low-on-the-SL. Today's #haskell exercise is write your own language, then some rules. Yeah. LAWLZ! NAND on the A-to-the-ST! A solution to the first 2 exercises to today's problem. YMMV: not pretty nor generic.
  • Baby needs a new pair of shoes! And you need a new daily #haskell problem to solve. Done! ;) Love is the Universal Language. No, wait. We were talking about Money, yes? Me confused. Solution to today's exercise I have a Yen for Thinking Japanese, or so I think so ... a solution to the bonus #haskell exercise
  • Bayes was a man of letters. A Bayesian classifier for letter-recognition? Sure, let's give it a go for today's #haskell exercise. So now we know 'M' looks like 'W' to some peeps but 'B', 'I', 'O' check out fine, so WE ARE READING WITH HASKELL! YAY ... added definitions to do test runs over the entire data set and then did various runs, tweaking the system. Results noted. Informative.
  • Today's #haskell exercise. An expert system for blood donation. I use the term 'expert' loosely, eh? ;) Have at it! "I vant to drinq yur bloot! Bwa-hahaha!" Okay. That quote is just ... creepy. A solution to today's exercise is at
  • Veni, Vidi, ... Duci? Today's #haskell exercise. It is a relief to know that no tactical nukes were launched by testing the solution to today's #haskell exercise. or How do you count Mississippi's in Roman numerals?
  • August 11th, 2014, Monday: You've heard of 'Rock Management'? How about ROCK LOBSTERS! GIVE ME A ROCK, NAOW!!!1!!!!11! Today's #haskell exercise 
  • August 12th, 2014, Tuesday: To Fract or Not to Fract: why are we even considering this question? Today's #haskell exercise 

  • Ooh! Bonus problem? On a Tuesday? Wowsers! Today's #haskell #bonus problem: "Continuum spectrum maximum... uh, 'thing'um"
  • August 13th, 2014, Wednesday. No problem! (literally), but: "Fract! The Cylons!" No... wait: Starbuck used a different word. A solution to yesterday's fracting problem is at
Flatliners. No fracting, no 'peak'ing, just a baseline data set of 
initializeSpectrum defined with flatline function.

Twin Peaks. Still not fracted, but data set with two spikes 
rolled into the base data set with smoothSpike fn 

Fracted. Data set with peaks, fracted using the frackedSpike function

  • August 14th, 2014, Thursday: Pining for that next fibo date-time before 2584. Today's #haskell problem inspired by @AlgebraFact Updated problem. Tightened up return value, removing unnecessary semideterminism. So, it gives, leik, a septatuple answer, leik. ... Yeah. Time keeps flowing like a (Fibonacci) river, to the sea. A solution to  the 'next Fibonacci' problem posted today.
  • Hail, Eris! or the Law of Fives or today's #haskell problem (implementing a Ripple-down rule set). Do it to it! One of three-parter solution to today's problem: Forall Existential(ism) A solution allowing forall-quantified values. Two of the three-part solution: All you need is fnord (la-di-da-di-dah)! Rippling down (and accumulating fired) rules Third of three-part solution: RippleDownRuleto ergo sum, adding rules to the RDR system  Updated the 3rd solution to the RDR (Writer monad definition) to ripple down to left and right, fixing inconsistency in rule findings.
  • August 18th, 2014: What's FP good for? Crypto and technical (financial) indicators. Why? Um, let's do today's #haskell exercise! Full on, all the way! Double Rain-... wait: line-graph. Solution to today's #haskell exercise

  • August 19th, 2014: What does it all mean? All the way! No, wait: this is just today's #haskell exercise (backtesting SMA indicator ). Take the monoid and run(State-WriterT), or WAAH! WAAH! I lost 100 corn chips on my investment strategy or, solution: But, as solace, it does come with a pretty picture, illustrating today's solution. Ooh! 

  • August 20th, 2014: Next up for today's #haskell exercise is the Exponential Moving Average. A solution to the E(xponential)M(oving)A(verage) #haskell problem: Stately Conformance 

  • August 21st, 2014: My Jeans. No ... I meant: 'GENES'! That's it. Today's #haskell exercise. A solution to (not even CLOSE to) 'yesterday's' #haskell exercise: Nature or Nurture? We present a genetic algorithm
  • August 22nd, 2014: Today's (much delayed) #haskell problem: the Silver Ratio (from @AlgebraFact Every cloud has a silver lining .. Every Haskell problem has a solution. (sometimes) (except noncomputable problems)

  • August 25th, 2014: LYADCFGG! Today's #haskell exercise? (automated/learned) Document classification.
  • August 26th, 2014: "Join me, Luke, on the Dark Side of the Force, and help me go #FORTH to solve today's #haskell exercise! MWA-HAHA!" The World's smallest DSL: Forth. A solution to today's #haskell exercise
  • August 28th, 2014: "As I compose this #haskell problem ... p.s. I love you ... You! You! You!" #1960s #song A little card logic problem. Oh, NOES! LOOK OUT! IT'S A CARD-#SHARKNADO! Nah, it's just 'today's #haskell solution, is all
  • August 29th, 2014: Groovitudes! WordNet Themes! or: today's #haskell exercise A 3x3 matrix has 36 possible solutions. A 5x5 matrix? Over 200M poss solutions? YIKES! A solution to the themed-words
  • The Forth language problem solution given on August 26th gives a very snazzy RPN ('reverse Polish notation') calculator, but that's all it does the ':'-defining word needs access and look-ahead to the program parsed stream, and that's a bit more to ask than pushing and popping stack operators.
  • For the August 29th problem(WordNet themes) the raw generated solution set is over 209M possibilities. My little Haskell program was not finished scanning them over four hours when I killed the process. However, my dear wife solved the problem in under five minutes. Setting aside the fact that she's a genius, the program needs to be better. It needs to use the ontology of English-language words to eliminate fruitless endeavors during their generation, not afterwards during the test phase.

Monday, August 25, 2014

Dylan: the harsh realities of the market

So, this is a little case study.

I did everything for Dylan. And when I say everything, I mean everything.  Here's my resumé:

  • I got excited about Dylan as a user, and I used it. I bought an old Mac that I don't ever remember the designation for, it's so '90's old, and got the floppies for the Dylan IDE from Apple research.
I'm not joking.

  • I integrated Dylan into my work at work, building an XML parser then open-sourcing it to the community under the (then) non-restrictive license. I think mine was the only XML parser that was industrial-strength for Dylan. Can't claim originality: I ported over the Common-LISP one, but it was a lot of (fun) work.
  • I made improvements to the gwydion-dylan compiler, including some library documentation (you can see my name right there, right in the compiler code), including some library functionality, did I work on the compiler itself? The Dylan syntax extensions or type system? I don't recall; if not in those places, I know I've looked at those guts: I had my fingers all over parts of the compiler.
I was in the Dylan compiler code. For you ll-types ('little language') that's no big deal.

But ask a software developer in industry if they've ever been in their compiler code. I have, too: I've found bugs in Java Sun-compiler that I fixed locally and reported up the chain.
  • I taught a course at our community college on Dylan. I had five students from our company that made satellite mission software.
  • I effing had commercial licenses bought when the boss asked me: what do we have to do to get this (my system) done/integrated into the build. I put my job on the line, for Dylan. ... The boss bought the licenses: he'd rather spend the $x than spending six weeks to back-port down to Java or C++.
  • I built a rule-based man-power scheduling system that had previously took three administrative assistants three days each quarter to generate. My system did it, and printed out a PDF in less than one second. I sold it, so that means I started a commercial company and sold my software.
I sold commercial Dylan software. That I wrote. Myself. And sold. Because people bought it. Because it was that good.

Hells yeah.

Question: what more could I have done?

I kept Dylan alive for awhile. In industry. For real.

So why is Dylan dead?

That's not the question.

Or, that question is answered over and over and over again.

Good languages, beautiful languages, right-thing languages languish and die all the time.

Dylan was the right-thing, and they (Apple) killed it in the lab, and for a reason.

Who is Dylan for?

That's not the question either. Because you get vague, general, useless answers.

The question is to ask it like Paul Graham answered it for LISP.

Lisp is a pointless, useless, weird language that nobody uses.

But Paul and his partner didn't care. They didn't give a ...


... what anybody else thought. They knew that this language, the language they loved, was built and designed and made for them. Just them and only them, because the only other people who were using it were college kids on comp.lang.lisp asking for the answers for problem-set 3 on last night's homework.

That's what Lisp was good for: nothing.
That's who Lisp was good for: nobody.

Same exact scenario for Erlang. Exactly the same. Erlang was only good for Joe Armstrong and a couple of buddies/weirdos like him, you know: kooks, who believed that Erlang was the right-thing for what they were doing, because they were on a mission, see, and nothing nobody could say could stop them nor stand against them, and all who would rise up against them would fall.


What made Lisp and Haskell and Erlang and Scala and Prolog (yes, Prolog, although you'll never hear that success story publicly, but $26M and three lives saved? Because of a Prolog system I wrote? And that's just one day in one month's report for data? I call that a success) work when nobody sane would say that these things would work?

Well, it took a few crazy ones to say, no, not: 'say' that it would work, but would make it work with their beloved programming language come hell or high water or, worse: indifferent silence, or ridicule, or pity from the rest of the world.

That is the lesson of perl and python and all these other languages. They're not good for anything. They suck. And they suck in libraries and syntax and semantics and weirdness-factor and everything.

But two, not one, but at least two people loved that language enough to risk everything, and ...

They lost.

Wait. What?

Did you think I was going to paint the rosy picture and lie to you and say 'they won'?

Because they didn't.

Who uses Lisp commercially? Or Haskell, except some fringers, or Scala or Clojure or Erlang or Smalltalk or Prolog

... or Dylan.

These languages are defined, right there in the dictionary.

Erlang: see 'career wrecker.'

Nobody uses those languages nor admits to even touching them with a 10-foot (3-meter) pole. I had an intern from college. 'Yeah, we studied this weird language called ML in Comp.sci. Nobody uses it.'

She was blown away when I started singing ML's praises and what it can do.

A meta-language, and she called it useless? Seriously?

Because that's what the mainstream sees.

Newsflash. I'm sorry. Dylan, Haskell, Idris: these aren't main-stream, and they never will be.

Algebraic types? Dependent types? You'll never see them. They're too ... research-y. They stink of academe, which is: they stink of uselessness-to-industry. You'll be dead and buried to see them in this form, even after they discover the eternity elixir. Sorry.

Or you'll see them in Visual Basic as a new Type-class form that only a few Microserfs use because they happened to have written those extensions. Everybody else?


Here's how Dylan will succeed, right now.

Bruce and I will put our heads together, start a company, and we'll code something. Not for anybody else to use and to love and to cherish, just for us, only for us, and it will blow out the effing doors, and we'll be bought out for $40M because our real worth is $127M.

And the first thing that Apple will do, after they bought us, is to show us the door, then convert the code into Java. Or Swift. Or Objective-C, or whatever.

And that's how we'll win.

Not the $40M. Not the lecture series on 'How to Make Functional Programming Work in Industry for Real' afterwards at FLoC and ICFP conferences with fan-bois and -girls wanting to talk to us afterwards and ask us how they can get a job doing functional programming.

Not that.

We'll win because we made something in Dylan, and it was real, and it worked, and it actually did something for enough people that we can now go to our graves knowing that we did something once with our lives (and we can do it again and again, too: there's no upper limit on the successes you're allowed to have, people) that meant something to some bodies. And we did that. With Dylan.


I've done that several times already, by my counting: the Prolog project, the Dylan project, the Mercury project, and my writing.

I'm ready to do that, again.

Because, actually, fundamentally, doing something in this world and for it ... there's nothing like it.

You write that research paper, and I come up to you, waving it in your face, demanding you implement your research because I need it to do my job in Industry?

I've done that to three professors so far. Effing changed their world-view in that moment. "What?" they said, to a person, "somebody actually wants to use this?" The look of bemused surprise on their faces?

It was sad, actually, because they did write something that somebody out there (moiself) needed, but they never knew that what they were doing meant something.

And it did.

Effing change your world-view. Your job? Your research? Your programming language?

That's status quo, and that's good and necessary and dulce and de leche (or decorum, I forget which).

But get up out of the level you're at, and do something with it so that that other person, slouched in their chair, sits up and takes notice, and a light comes over their face and they say, 'Ooh! That does that? Wow!' and watch their world change, because of you and what you've done.

Dylan is for nothing and for nobody.

So is everything under the Sun, my friend.

Put your hand to the plow, and with the sweat of your brow, make it yours for this specific thing.

Regardless of the long hours, long months of unrewarded work, and regardless of the hecklers, naysayers, and concerned friends and parents, and regardless of the mountain of unpaid bills.

You make it work, and you don't stop until it does.

That's how I've won.

Every time.

Friday, August 1, 2014

1HaskellADay July 2014 problems and solutions

  • July 1st, 2014: (text:) "Hi, all! @1HaskellADay problems are back! #YAY First (renewed) problem: a verbose way of saying, 'Give me a deque!'" Deque, last, and all that (verbose version with hints) (solution: Deque the halls (with my solution): Data.Deque)
  • July 2nd, 2014: (text:) "Today's Haskell exercise: Vectors, length in constant time, and (bonus) reverse return in constant time." Vector (solution: Vector: Magnitude, and Direction, OH, YEAH! Data.Vector)
  • July 4th, 2014: (text:) "Today's exercise(s). pack/unpack. encode/decode. Cheer up, sweet B'Genes!" Cheer up, Sweet B'Genes (solution: GATTACA)
  • July 7th, 2014: (text:) "#haskell daily exercise: ROLL with it, Baby!  ('cause I'm feeling a little #forth'y')" Roll (solution: Rollin' on the (finite) river)
  • Bonus problem: July 7th, 2014: (text:) "For those who found the 'roll'-exercise trivial; here's (a more than) a bit more of a challenge for you to play with." Acid rules! (solution: "A solution set to today's challenge exercise: BASIC ...  ... and Acidic  ... WHEW! That was fun!" BASIC ... and Acitic)
  • July 8th, 2014: (text:) "Today's #Haskell exercise: LOTTO! Powerball! Mega-whatever! Who's the big winner?" Lotto (solution: "And the big winner today is ... solution-set to today's #Haskell lotto exercise" ... and the winner is ...)
  • Bonus problem: July 8th, 2014: (text:) "#bonus #haskell exercise: Well, that was RND... Randomness, and such (or 'as such')" Well, that was RND (solution: For YESTERDAY's bonus question of roll-your-own-rnd-nbr-generator, here's one as comonadic cellular automata (*WHEW*) Data.Random)
  • July 9th, 2014: (text:) "Okay, ... WHERE did yesterday and today GO? :/ #haskell exercise today: "Hey, Buddy!"  I will post solution in 4 hours" Hey, Buddy! Distinct sets-of-an-original-set. (solution: "Here's a story ..." A(n inefficient) solution to bunches and cliques." Brady Bunch)
  • July 10th, 2014: (text:) "Today's #haskell list-exercise: "Get out of the pool!"  Will post a solution at 9 pm EDT (which is what time CET? ;)" (solution: "She's a supa-freak! She's supa-freaky! (Bass riff) A solution to today's #haskell exercise about list-length-ordering")
  • July 11th, 2014: (text:) ""It's Friday, Friday!" So, does that mean Rebecca Black wants to code #haskell, too? Today is a Calendar #exercise" (solution: ""In a New York Minute": a solution to today's #haskell exercise that took WAAAY more than a minute to complete! #WHEW")
  • July 14th, 2014: (text:) "Today's #haskell exercise: isPrime with some numbers to test against. They aren't even really Mp-hard. ;)" First stab at primality-test (solution: "A simple, straightforward stab at the test for primality. #haskell #exercise" The start of a primal inquiry
  • July 15th, 2014: (text:) "Primes and ... 'not-primes.' For a prime, p, a run of p-consecutive 'plain' numbers is today's #haskell exercise:" (solution: "So ya gotta ask yerself da question: are ya feelin' lucky, punk? Run of p non-primes in linear time  #haskell exercise." Alternate solution by Gautier:
  • July 16th, 2014: (text:) "Difference lists? We no need no steenkin' Difference lists!"  DList in #haskell for today's exercise. (solution: "DLists? We got'cher DList right here! A solution to today's #haskell exercise is posted at")
  • July 17th, 2014 (text:) " Prélude à l'après-midi d'un Stream ... I thought that last word was ... something else...? #haskell exercise today." Comonads for lists and Id. (solution: "Control.Comonad: That was easy! #haskell exercise #solution" Learn you a Comonad for Greater GoodFunny story, bro'! id is not necessarily Id. (I knew that.) #haskell solution")
  • Bonus exercise: July 17th, 2014 (text:) "Streams are natural, streams are fun, streams are best when ... THEY'RE BONUS QUESTIONS! #bonus #haskell exercise" LET'S GET THIS PARTY STARTED! (solution: "Take this Stream and ... it! #solution to today's #haskell #bonus exercises")
  • July 18th, 2014: (text: "Today's #haskell exercise: Frère Mersenne would like a prime, please.") (see solution next bullet)
  • Bonus exercise: July 18th, 2014 (text: "#bonus prime-time! Frère Mersenne would be pleased with a partial proof of a prime ... in good time.") (solution: "A #haskell #solution for (monadic?) primes and the #bonus interruptible primes.") Primary primes.
  • Bonus-bonus exercise: July 18th, 2014 (text: "Ooh! π-charts! No. Wait. #bonus-bonus #haskell exercise.") (solution: "#bonus-bonus: a #solution")

  • July 21st, 2014: (text: "Demonstrating coprimality of two integers with examples #haskell exercise") (solution: "A coprimes solution #haskell problem is at")
  • July 22nd, 2014: (text: "The prime factors of a number (and grouping thereof) as today's #haskell exercise.") (solution: "OKAY, THEN! Some prime factors for ya, ... after much iteration (torquing) over this #haskell exercise solution.")
  • Bonus exercise: July 22nd, 2014: ("For today's #bonus #haskell exercise you'll find a Bag 'o gold at the end of the rainbow") (solution: "Second things first: a definition for the Bag data type as today's #bonus #haskell exercise.")
  • July 23rd, 2014: (text: "Today's #haskell exercise, two variations of Euler's totient function") (solution: "And, for a very small φ ...  is a solution-set to today's #haskell exercise.")
  • July 24th, 2014: (text: "WEAKSAUCE! Goldbach's conjecture irreverently presented as a #haskell exercise.") (solution: "That solution to today's #haskell exercise will cost you one Goldbach (*groan!*)")
  • July 25th, 2014: LOGIC! Peano series: it's as easy as p1, p2, p3 ... ... in today's #haskell exercise. "Excuse me, Miss, where do we put this Grande Peano?" A solution to today's #Haskell exercise in the HA!-DSL
  • Bonus: July 25th, 2014: Bonus #haskell problem for today. But not as easy as λa, λb, λc ... Church numerals and booleans. Correction: Ooh! forall! Church encodings and Haskell have a funny-kind of relationship. Updated the #bonus #haskell exercise with rank-types and forall. Solution: "Gimme that olde-time Church encoding ... it's good enough for me!" A solution to today's #bonus #haskell exercise
  • July 28th, 2014: George Boole, I presume? Today's #haskell exercise: Solution: This and-or That ... a NAND-implementation of today's #haskell exercise at
  • July 29th, 2014: Readin' 'Ritin' 'Rithmetic: today's #haskell exercise Solution: That's alotta NANDs! A solution to today's exercise at
  • July 30th, 2014: ACHTUNG! BlinkenLights! Today's #haskell exercise Solution: Let it Snow! Let it Snow! Let it (binary) Snow! A solution to today's exercise is at 
  • July 31st, 2014: π-time! Today's #haskell exercise. BLARG! UPDATE! Please read the update attached to the problem statement, simplifying the calculation quite a bit: Solution: Apple or coconut π? A solution to today's problem

Notes on the problems
  • July 9th, 2014. I didn't quite know how to go about this, so I made several attempts with the State pattern. But how to describe it? It's the base pool from which you draw, and each (sub-)choice-point affects it, what's that type? I spent way too much time trying to discern the type, and failing. But now, a much simpler approach suggests itself to me (after experiencing the New York Minute exercise): this is simply a permutation of the list, and then that permutation is partitioned by the sizes of the groups! Implementing permute-then-partition is a much simpler approach than tracking some monster monadic state transformer.

    No, that didn't work, either. A permutation will give you [[1,2], ...] and[[2,1], ...] That is, all solutions, even the redundant ones. So, I reworked the problem simply following the data. With takeout feeding the iterative-deepening function, I finally got a guarded state-like thingie working fast and correctly. The new solution is on the same page as the old one.
  • July 11th, 2014. The New York Minute problem demonstrates the implementation of a rule-based classifer. It takes unclassified numeric inputs, and based on the cues from the rules, either types each number into day, month, year, hour, minute, or rejects the input data as unclassifiable. I was pleased to have implemented this entire system in less than two days of work! Sweet!
  • July 22nd, 2014. So, I've been running up against the double-exponential cost of computing a stream of primes for some time now since I gave the solution to the question of demonstrating the Prime Number Theorem. So now I have to tackle of bringing down that extraordinary, or unreasonable, cost down to something useable, and along those lines (of feasibility), I'm thinking of instead of regenerating and re-searching the primesish stream that we have some kind of State-like thing of ([already generated primes], indexed-primesish) ... something like that. Solution: "General improvement of problem-solving modules in anticipation of solving today's #haskell exercise, including primes:"

Monday, July 28, 2014

Are you serious? "What degree?"

So, serious question.

An aside (get used to them, by the way), people are never sure if I'm serious.

I'm, like, seriously? C'mon!

I am always silly (well, oftentimes silly, compared to my dour workmates), but I am always serious in my silliness.

Chesterton said it well: the opposite of funny is not serious: the opposite of funny is not funny.

Okay, that aside is done. Now onto the topic at hand.

Hands up, those who have ever used your education in your jobs. Hands up, those who needed proof of your degree to prove competency as a prerequisite of employment.

(sound of crickets.)

Thought so.

Actually, more of you might raise your hands to me than to most of my colleagues, because why? Because I have close ties to academe, that's why. So there are more than a few of you who are required to have your Master's degree or, those of you who are post-docs, to have your Ph.D. to get that research position or grant that you're working on.

The rest of the world?


Education, in the real world, is a detriment to doing your job.

Across the board.


Do you know how many Ph.D.s we fire? Do you know how many Ph.D.s and matriculated students we turn away, because of their education and their lack of real-world experience?

We did a survey: a sure bell-weather for a prospective employee? The amount of their education: the more they have, the more likely they are to be useless on the job.

It's the Ph.D-disease: 'Ph.D: "piled high and deep."' People get ed-ju-ma-kated and then they think because they have a sheepskin or that they have a certain GPA at a certain school and know a certain ... educated way of doing things, that they know how to do this-and-that other thing, totally unrelated on the job.

"I've studied the Martin-Löf's intuitionistic type theory."

"Great, how do you connect to our database."

"Uh ...."


You'll bridle at this, but you know who agrees most strongly with me?


I went to the ICFP2006 specifically looking to roll dependent types into the programming language I was using, in industry, and I could not get the time of day from a single implementer of the type-theory in Twelf, and you know why?

"Why would you be interested in this? This is purely theoretical."

Uh, huh. As in "not applicable to industry."

I reread the paper again, later: "Dependent types make programs easier to understand and more reliable."

And why would I want that in my programs in industry, where it mattered.

I spent four years at the United States Coast Guard Academy: 10,000 students apply, only 300 are allowed in, only 150 graduate, each year. It cost, at the time, $250,000 to graduate a student from the Academy, making it the most expensive school in the nation.

Thank you, taxpayers, for my education.

I graduated with a dual major: mathematics and computer science.

How much of my education did I use on the job to save 150 lives from a capsized Arctic research exploration vessel (boy, they surely used their education, didn't they! ... to capsize their ship), act as the translator when we boarded Japanese trawlers, provide civil rights education and mediate EEO complaints and ...

None. Zip. Zilch.

After my stint, how much of my college education did I use on the job.

My job was encoding matrices in FORTRAN. How much FORTRAN did I study in college?

Zip. Zilch. Nada.

How much of the Advanced Calculus II did I use on the job.

People, it was frikken matrix manipulation! Something you can (now) look up on wikipedia and pick up in, oh, two hours if you're really slow.

Java. C#. Swift. Visual whatever. Spring. SQL. HBase. Angular. JavaScript.

All these things (Python, Ruby on Rails) can be taught in college, but they can be taught in high school, and I learned them in school, did I?

No, I did not. I learned them on my own, thank you very much.

Design patterns, frameworks, data structures. Do educated people know these things?

Some of them do. Most people with a 'computer science' degree DO NOT, people.

They do not. They woefully do not, as comp sci teachers lament over and over again, and as I, the hiring manager, scratch my head wondering what, precisely, did these kids learn in school, because, insofar as I see, they did not learn abstraction, polymorphism, typing, or data structures.

They learned the if-statement.

They learned the if-statement that n-dispatches within a for-loop from 1 to n off an array. That's the data structure they know: the array.

Maps? Sets?

The array.


We got you covered: the if-statement.

Functional decomposition?

Well, there's always main(String[] args), with a big-ole if-statement.

The word on the street is education is a canard at best and a detriment at most, and at worst, it's a project-sinker.

That's a shame, because there are educated people who are productive and smart and effective in their field, and can help.

How to Solve It: a Modern Approach claims that one billion dollars is wasted on software because it's written in the absence of very simple techniques, such as linear programming.


Our authors, Michalewicz and Fogel, are off. Way off. I know. By a factor of at least one-hundred.

We wasted a billion dollars on a file-indexing system for the FBI. Oh, it had email, too. Government project. Never used because it was never delivered in a useable state. Do you know how many go through that cycle?

I don't. I do know know I've seen project after project just ...

God. The waste.

And I have friends. And they tell me stories.

But, you get MITRE in there, or you get a Ph.D. or two in there, and what happens?

They study the issue.

They study it for six-plus months, and then they write you a nice little white paper that states the obvious search criteria that you knew from day one, but what do you have to say? Where is your ROC-analyses? So your bayesian system that was cranking out results month after month was killed by the bean-counting pointy-heads and they submit a ... working solution that could go into production? Oh, no. They submit a white paper calling for a research grant to allow for a year of surveying and further study of the issue.


Then they get fired or they move on to more interesting research areas leaving behind us to clean up the mess and get a working system out the door in some serviceable shape that used zero percent of their research.

Zero percent.

You see, they modeled the situation, but the model doesn't fit the data, which is raw and dirty, so their solution solved the model, not your problem, not even close.

Your degree.

How much have you used your degree on your job?

If you're a researcher, you probably use quite a bit of what you've studied in your research, and you are contributing more to your field of study.

If you're not, then you're told this, day one, on your job: "Friend, put those books away, you're never going to use them again."

I mean, seriously: did you really bring your college books to your job thinking you'd use them?


This, here, is the real world. The ivory tower is for the academics. In the real world, you roll up your sleeves, get to work, and get some results; because if you don't, the door is right over there.

You were expecting to use your degree on your job? This is America, people. We don' need no edjumakashun.

Now, if this were Soviet Russia, your degree uses you.


So, silliness, and serious silliness aside.


You were expecting to use your degree on your job?

English major. This is America, we don't talk English, we talk American, so, that's nice that you have that degree.

Mathematics major. This is America, we don't do 'maths,' nor trig, nor geometric algebras, nor category theory, how would you use any of that on your job?

I was seriously asked that on my interview for a job overseeing a 1.7 petabyte-sized database.

I said: 'uh, map-reduce are straight from category theory.'

"Yes, but how do you use that on your job?"

We both blinked at each other dumbly.

The gulf.

You don't go to school to get trained to do a job well, ladies and gentlemen.

I mean, too many of you do that, and too many others go do school to party some, to sex some, to blaze some, and then get to work after your folks financed your four-plus year bacchanal.

College is not a technical training-institute and has nothing to do with acquiring skills or proficiency on repetitive stress disorder, oh, I meant: 'your job.' Your job, almost without exception, can be just as proficiently performed by nearly anyone they drag off the street and put in your chair for eight hours a day. They sit in your chair for a few days, and everyone else won't even know you're gone.

Most jobs.

My wife beat the pants off her entire payroll division with an excel spreadsheet because they didn't have simple accounting principles and deductive reasoning. Why? Because they were well-regulated at their jobs, proficient at it, in fact, and their job was to make continuous clerical errors because they had absolutely no rigor. Why would they? They weren't paid for rigor. They were paid for doing their jobs, which was: don't make waves.

I regularly go into situations where other software engineers (a misnomer, they are more like computer programmers, not engineers) say such-and-so cannot be done in programming language X.

Then, I implement a little bit of category theory, in programming language X, do some simple mappings and natural transformations, and, voilà! those 50,000 lines of code that didn't solve the problem but only made things worse? I replace all that with 500 lines of code that actually delivers the solution.

Unit tested: all the edge cases.

And meeting their requirements, because I've translated the requirements into a declarative DSL on top of their programming language X.

Of course they couldn't solve the insurmountable problem in programming language X, not because they were using programming language X (although it helped with the colossal fail being object-disoriented and improvably/mutatatively impure), but because they couldn't think outside the box that 'you can only do this and that' as a software engineer. They were caught in their own domain and can't even see that they had boxed themselves in.

Because they were educated that way. Comp Sci 101: this is how you write a program. This is the 'if'-statement. This is the for-loop. If that doesn't work, add more if-statements wrapped by more for-loops, and this statement is perfectly acceptable:

x = x + 1

Go to town.

That's what their education gave them: they went to school to acquire a trade and a proficiency at the if-statement, and gave up their ability to see and to think.

And some, many, academics are the most bigoted, most blundering blinders-on fools out there, because they see it their way, and they see their way as the only way, which requires a six-month research grant and further study after that.

With co-authorship on the American Mathematical Society journal article.

And the uneducated are the worst, most pigheaded fools out there, so sure that the educated have nothing to offer, that they have no dirt under their perfectly manicured fingernails attached silky smooth hands that have never seen an honest-day's work nor, God forbid! a callous, so what do they know, these blowhards, so the uneducated ignore the advances of research into type-theory, category theory, object theory (polymorphism does help at times), any theory, and just code and code and code until they have something that 'looks good.'

How to solve this?

Start with you.

Not with your education, that is: not with your education that tells you who you are.

Start with how you can help, and then help.

  • Project one: I saw how fractal dimensionality would solve a spectrum analysis problem. Did I say the words 'fractal' or 'dimensions'? No. I was working with real-programmers. If I asked if I could try this, do you know what they would say?

    Pfft. Yeah, right. Get back to work, geophf!

    But, instead, I implemented the algorithm. I sat with a user who had been working on those signals and knew what he needed, iterated through the result a week.

    Just a week. While I did my job-job full time. I did the fractal spectrum analysis on my own time.

    My 'thing' floored the software management team. They had seen straight-line approximations before. They thought I was doing actual signal analysis. I mean: with actual signals.

    They showed my 'thing' to the prospective customer. And got funded.

  • Another project: data transformation and storage, built a system that encompassed six-hundred data elements using a monadic framework to handle the semideterminism. That was an unsolvable problem in Java.

    I used Java.

    Java with my monadic framework, yes, but Java, to solve the problem.

  • Third project: calculating a 'sunset date' over a data vector of dimension five over a time continuum.

    Hm: continuum.

    Unsolvable problem. Three teams of software developers tackled it over six months. Nobody could get close to the solution.


    I used a comonadic framework.

    Took me, along with a tester who was the SME on the problem, and a front-end developer to get the presentation layer just right, about a month, and we solved that baby and put it to bed.

    Unit tested. All edge cases.

    Did I tell them I used a comonadic framework?

    Nah, they tripped over themselves when they saw the word 'tuple.'

    No joke, my functional programming language friends: they, 'software engineers,' were afraid of the word 'tuple.'

    So I explained as much as anyone wanted to know when anyone asked. I wrote design documents, showing unit test case results, and they left me alone. They knew I knew what I was doing, and I got them their results. That's what they needed.

    They didn't need my degree.

    They didn't need to know I used predicate logic to optimize SQL queries that took four hours to run to a query that took forty-five seconds.

    They didn't need to know I refactored using type theory, that A + B are disjoint types and A * B are type instances and A ^ B are function applications so I could look at a program, construct a mathematical model of it and get rid of 90% of it because it was all redundantly-duplicated code inside if-clauses, so I simply extracted (2A + 2B ... ad nauseam) to 2(A + B ...) and then used a continuation, for God's sake, with 'in the middle of the procedure' code, or, heaven help me, parameterization over a simple functional decomposition exercise to reduce a nightmare of copy-and-paste to something that had a story to tell that made sense.

How do you connect to a database?

Do you need a college degree for that?

Kids with college degrees don't know the answer to that simple interview question.

And they don't know the Spring framework, making 'how to connect to a database' a stupid-superfluous question.

They don't know what unit tests give them. They don't know what unit tests don't give them. Because they, college kids and crusty old 'software engineers,' don't write them, so they have no consistency nor security in their code: they can't change anything here because it might break something over there, and they have no unit tests as a safety net to provide that feedback to them, and since they are programming in language X, a nice, strict, object-oriented programming language, they have no programs-as-proofs to know that what they are writing is at all good or right or anything.

A college degree gives you not that. A not college degree gives you not that.

A college degree is supposed to what, then?

It's suppose to open your mind to the possibility of a larger world, and it's supposed to give you the tools to think, and to inquire, so that you can discern.

"This, not that. That, and then this. This causes that. That is a consequence of this. I choose this for these reasons. These reasons are sound because of these premises. This reason here. Hm. I wonder about that one. It seems unsound. No: unfamiliar. Is it sound or unsound? Let me find out and know why."

English. Mathematics. Art. Literature. Music. Philosophy. All of these things are the humanities. The sciences and the above. Law. Physics. All these lead one to the tools of inquiry.

In school, you are supposed to have been given tools to reason.

Then, you're sent back out into the world.

And then you are supposed to reason.

And with your reason, you make the world a better place, or a worse place.

These things at school, these are the humanities, and they are there to make you human.

Not good at your job, not so you can 'use' your degree as a skill at work, but to make you human.

And, as human, are you good at your job?


And, as human, do you make your world a place such that others are good and happy at their jobs?


The end of being human is not to be skilled, nor proficient ... 'good' at your job.

But it's an accident of it, a happy accident.

The 'end' of being human?

Well: that's your inquiry.

That's what school, that's what everything, is for: for you to answer the unanswered question.

Your way.

And, if you accept that, and are fully realized as a human being, then your way is the best way in the world, and your way has the ability to change lives. First, your own, then others. Perhaps your coworkers.

Perhaps hundreds of others.

Perhaps thousands.

Perhaps you will change the entire world.

But you won't know that until you take that first step of inquiry.

Then the next.

Then the next.

And you look back, and you see how far you've come, and ... wow.

Just wow.

That's what school is for. Not for your job.

For you.

Saturday, July 12, 2014

1HaskellADay: Up, up, and away!

I've taken it upon myself to submit problems, then show the solutions, for @1HaskellADay. I started this work on July 1st, 2014. So the next set of entries serve to collect what I've done, both problems and solutions, and, if a particular day posed a particularly interesting problem, that I solved, or, that I didn't solve, I'll capture that here, too.

Monday, June 23, 2014

matchingSub is Comonadic (obviously!)


Today’s 1HaskellADay problem was an interesting NP-hard problem ... until I reread the problem statement carefully, then it became trivial. ‘Consecutive’ is the keyword here that unlocked the puzzle for me, eventually.

The heart of the algorithm here is simple enough. It says, in words, ‘are these next n numbers in the list the sum specified?’ If they are, return them as a solution, if they are greater, return nothing, if they are less than the sum, keep trying.

Those are the words, here is the implementation:

inquiry :: Int -> Int -> DList Int -> [Int] -> [Int]
inquiry _ _ _ [] = []
inquiry goal sub accum (next : rest) =
   let tote = sub + next
       naccum = accum << next
   in  case (compare tote goal) of
          LT -> inquiry goal tote naccum rest
          EQ -> dlToList naccum
          GT -> []

Simple enough, and I use a difference list as an accumulator, just because that’s how I roll (and the fact, too, that difference lists append an element in constant time is sweet!)

So, now all we need to do is move an index over the list to test each possible scenario.

Enter the (Comonad) Dragon

Of course, iterating over a list, keeping the context of the list itself active can be done functionally in many ways, but what suggested itself to me right away was the Comonad.

The Comonad of the list is the list itself and all of its tails, and this is exactly the paradigm we need to solve this problem simply, so, writing matchingSub became simply a comonadic extension:

matchingSub’ :: Int -> [Int] -> [[Int]]
matchingSub’ goal domain =
   domain =>> inquiry goal 0 emptyDL

Now, this doesn’t quite give the requested solution sets for the given examples, as it returns the empty list as ‘no solution,’ not ‘nothing at all’ as requested, but filtering out the empty list is easy enough: we just need a predicate to test if the list is not empty and then return only the lists of answers:

isCons :: [a] -> Bool
isCons [] = False
isCons _ = True

matchingSub :: Int -> [Int] -> [[Int]]
matchingSub goal = filter isCons . matchingSub’ goal

And there we have it!

I like comonads. They’re sweet!

Okay, prove it, buster!

(Who is this 'buster'-person, by the way, and why is he always having to prove things?)

All the above code is in Haskell programming, but it is also mutually-translatable to and from Idris. Nearly the same syntax (intentionally so), and nearly the same semantics (Idris's eager evalution looks and tastes very much like Haskell's normal order evaluation).

In Haskell, we'd hand-verify the above with the provided samples and we're done. We'd run it through quickcheck to be done-done.

In Idris, we can prove that what we specified is actually (heh: 'actually') correct in its implementation.

So, let's prove it.

prvSample : (expected : List (List Int)) 
            -> (actual : List (List Int))
            -> (so (expected == actual)) -> ()
prvSample expected actual pred = ()

Using the 'so' assertion, we're (almost) done.

Let's run our sample-set through our prover, such that it is:

sample1 : ()
sample1 = (prvSample [[1..4]]
                     (matchingSub 10 [1..5])) oh

sample2 : ()
sample2 = (prvSample [[1,1], [1,1]] 
                     (matchingSub 2 $ replicate 3 1)) oh

sample3 : ()
sample3 = (prvSample [[1,1], [1,1]]
                     (take 2 $ matchingSub 2 $ repeat 1)) oh

Code compiles? Yes? We're done: we've delivered functionality as specified.


Saturday, June 21, 2014

keepEquals with Difference Lists

Yesterday's 1HaskellADay exercise was keepEqual but it could have easily have been ‘keepSimple’ or ‘write this in your sleep’ or ... something like that.

It was a good exercise, because there’s the obvious solution but then you look to improve the efficiencies.

And then, of course, there’re the wrong solutions, but that’s what we have proofs for, to prove ourselves into the correct implementations.

So the first stab I did was wrong:

keepEqual1 : Eq a => List a -> List a -> List a
keepEqual1 list1 list2 = list1 >>= \h1 =>
                         list2 >>= \h2 =>
                         (if h1 == h2 then return h1 else [])

The problem here is that this algorithm, albeit cleverish is, well: wrong.

It iterates through each element of list1, fine, but it compares the currently-selected element of that list to every element of list2.  There’s also a tiny problem in that if either list1 or list2 are infinite, see, as the binding operation goes through each element of the list (inner, then outer) before returning, which could be a problem if you ever want a return in this eternity.

Minor problem there.

The other problem is that this is an exponential algorithm: for each element of list1, it possibly iterates through the entire set of list2 to find a match.


So. Cleverish approach was a fail on my part. Shall we try the traditional approach?

keepEqual2 : Eq a => List a -> List a -> List a
keepEqual2 [] _ = []
keepEqual2 _ [] = []
keepEqual2 (h1 :: t1) (h2 :: t2) =
   (if h1 == h2 then [h1] else []) ++ (keepEqual2 t1 t2)

So, that’s traditional. That works (and we can choose to verify that it does work), and it terminates at the end of the first list, thereby neatly dodging the non-termination-with-infinite-list-arguments issue.

The problem here is that we are representing choice with a list-compressionescque algorithm here, so we continuously concatenate to the end of a single-element list, or, in the case of a non-match, the empty list.

That algorithm, concatenation-no-matter-what, just screams: “Improve me! Improve me, please!”

So, okay, one improvement is we can turn our choice-point from the above to construction or return:

   (if h1 == h2
    then (h1 :: (keepEqual2 t1 t2))
    else keepEqual2 t1 t2)

Okay, is that yummy-looking?

No. No, it is not.

I mean, for efficiency’s sake we are eliminating the whole ‘concatenate after the empty list’ operation for the not-equals case, and keepEqual2 is being called only once in this branch, but ...

But it’s still written twice, and that’s ugly. Why do we have to express something twice for the concept of ‘okay, add the head only if it’s the same and then continue on with the rest of the two lists.’ I mentioned the continuation (ooh! continuations!) just once here in my denotation, why do I have to mention it twice in my implementation?

Well, I don’t have to, actually. We’re working in a functional domain, so let’s get functional!

   (if h1 == h2 then ((::) h1) else id) (keepEqual2 t1 t2)

Boom! ... actually, not so ‘boom’ because of the overridded values of (::) confuses Idris (Vect, Stream, and List all use (::) as a constructor), so let’s clarify that:

cons : a -> List a -> List a
cons a list = a :: list

So, now we can write:

   (if h1 == h2 then (cons h1) else id) (keepEqual2 t1 t2)

I actually had this little partial function revelation just now, so my actual implementation involved me creating the difference list data type, which allowed constant time prepend and append operations. Which is better? Partial functions or the difference list?

Well, let’s take a look at my difference list implementation so we can judge their respective merits.

data DList a = DL (List a -> List a)

A difference list is the difference between two lists, and we represent this difference by capturing it in a function

unDL : DList a -> (List a -> List a)

And we can, of course, extract our function from the difference list type.

(<<) : DList a -> a -> DList a
list << a = DL ((unDL list) . (cons a))

Our append operator (<<) appends an element to the tail end of a DList in constant time (if it were needed, the corresponding (>>) operator prepends an element to the head of a DList, also in constant time).

So, we have our DList structure, but we need to start from something, and then, eventually convert back to a plain-old List structure. The below functions provide those functionalities for us:

emptyDL : DList a
emptyDL = DL id

dlToList : DList a -> List a
dlToList dlist = with List (unDL dlist [])

Now, with the difference list type, we rewrite our keepEqual in the natural style.

keepEqual : Eq a => List a -> List a -> List a
keepEqual’ : Eq a => DList a -> List a -> List a -> List a

keepEqual = keepEqual’ emptyDL

keepEqual’ dl [] _ = dlToList dl
keepEqual’ dl _ [] = dlToList dl
keepEqual’ dl (h1 :: t1) (h2 :: t2) =
   keepEqual’ (if (h1 == h2) then dl << h1 else dl) t1 t2

So, what do you think? For this example the DList doesn’t really shine all that much, but if we had something where appending to the end (in constant time) made a whole lot of sense, then we would have a clear win for it.

I like DList.