Quit

Neural networks

Building a simple neural network to break a CAPTCHA such as the one shown below:

SetDirectory@NotebookDirectory[];
Example=Import@"captcha.jpg"

Preliminaries

The CAPTCHAs we are dealing with use the characters a-z0-9

CHARS=Join[Alphabet[],CharacterRange["0","9"]]

We load the datasets as lists of rules x->y, where y=f(x) is the function we want to learn

ImportDataSet[files_]:=Table[Import@file->StringTake[FileNameTake@file,8],{file,files}]

TrainingData=ImportDataSet@FileNames["train/*.jpg"];
TestData=ImportDataSet@FileNames["test/*.jpg"];

Pre-processing

We will train a neural network to solve a simpler problem, namely the mapping from each of 8 pieces of a CAPTCHA to the corresponding (single) character they show.
We discard color information, since it won’t help in this case.

SplitImage[im_]:=With[{newim=ColorSeparate[ImageTake[im,{13,44},{20,-18}],"V"]},
Table[ImageTake[newim,All,{x,x+24}],{x,0,140,20}]
]

SplitImage@Example

We will encode the 36 characters in our alphabet with 36-dimensional one-hot vectors. This is required for Mathematica to be able to guess an appropriate loss function for our network.

Encoder=NetEncoder[{"Class",CHARS}];
Encoder@"f"

Finally, we prepare the new dataset again in the form x->y

RandomShuffle[l_List]:=RandomSample[l,Length@l]

PrepareDataSet[dset_]:=RandomShuffle@Flatten@Table[First@#->Last@#&/@Transpose@{SplitImage@First@dpoint,Encoder/@Characters@Last@dpoint},{dpoint,dset}]

PreppedTrainingData=PrepareDataSet@TrainingData;
PreppedTestData=PrepareDataSet@TestData;
PreppedTestData[[;;10]]

PreppedTrainingData[[;;10]]

Defining and training a neural network

We will use the following neural network (based on LeNet-5)

NeuralNetwork=NetChain[{
ConvolutionLayer[20,{5,5}],
ElementwiseLayer@Ramp,
PoolingLayer[{2,2},{2,2}],
ConvolutionLayer[50,{5,5}],
ElementwiseLayer@Ramp,
PoolingLayer[{2,2},{2,2}],
FlattenLayer[],
LinearLayer[500],
ElementwiseLayer@Ramp,
LinearLayer[36],
SoftmaxLayer[]
},
"Input"->NetEncoder[{"Image",{25,32},"Grayscale"}],
"Output"->NetDecoder[{"Class",CHARS}]
]

TrainedNeuralNetwork=NetTrain[NeuralNetwork,PreppedTrainingData,MaxTrainingRounds->50]

The accuracy of our network seems great, but remember we need to get 8 guesses right for a single CAPTCHA to be broken.

NetMeasurements[TrainedNeuralNetwork,PreppedTestData,"Accuracy"]

%^8

Streamlining the solution

Define our SolveCAPTCHA function and evaluate the real accuracy

SolveCAPTCHA[im_]:=StringJoin[TrainedNeuralNetwork/@SplitImage@im]
SolveCAPTCHA@Example

EvaluateAccuracy[solver_,dset_]:=Boole/@Table[solver@First@dpoint==Last@dpoint,{dpoint,dset}]//Mean//N

EvaluateAccuracy[SolveCAPTCHA,TrainingData]

EvaluateAccuracy[SolveCAPTCHA,TestData]

Playing with built-in Mathematica functions

PrepareDataSet2[dset_]:=RandomShuffle@Flatten@Table[First@#->Last@#&/@Transpose@{SplitImage@First@dpoint,Characters@Last@dpoint},{dpoint,dset}]

Classifier=Classify@PrepareDataSet2@TrainingData

ClassifySolver[im_]:=StringJoin[Classifier/@SplitImage@im]

EvaluateAccuracy[ClassifySolver,TestData]

Using a massive dataset of (10^4) CAPTCHAs

Not much improvement using 10x the number of training examples...

MassiveTrainingData=ImportDataSet@FileNames["C:\\Users\\fidel\\Desktop\\massive\\*.jpg"];
MassiveTestData=ImportDataSet@FileNames["C:\\Users\\fidel\\Desktop\\Dropbox\\Programacion\\Prenota\\captcha\\data\\labelled\\test\\*.jpg"];
Length/@{MassiveTrainingData,MassiveTestData}

TrainedNeuralNetwork2=NetTrain[LeNet,PrepareDataSet@MassiveTrainingData,MaxTrainingRounds->50]

NetMeasurements[TrainedNeuralNetwork,PrepareDataSet@MassiveTestData,"Accuracy"]

NetMeasurements[TrainedNeuralNetwork2,PrepareDataSet@MassiveTestData,"Accuracy"]

EvaluateAccuracy[SolveCAPTCHA,MassiveTestData]

SolveCAPTCHA2[im_]:=StringJoin[TrainedNeuralNetwork2/@SplitImage@im]

EvaluateAccuracy[SolveCAPTCHA2,MassiveTestData]