Download data

Dataset 1 Link : https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

Dataset 2 Link : https://www.gutenberg.org/files/2600/2600-0.txt

Using Only Postion Encoding with Linear Transformation

et in the prestine and slow movements of a small town. The sun had just begun to set over the quiet town, casting long shadows across the narrow streets. The air was still, and there was a strange feeling that something was about to happen. t Dowhem, thecrse athadd itho co tus ine woxpe kn. ot lderimpore hermpiser “Borirsthrted hockelatok hothecherend agitolldd, fir, lan hang whermaly!” The wheno—incomelarelad Whekste dyour bul “Whta: ad t fatuershin to ismowasthe anthe inofiow. t She ite l?” terissted ur.” wokenouispples os adofiric, w imu us t erg Aricon m. ofuar her o Buroen as tt sole t fens m.

Using Single-Attention Head and FeedForward Network.

et in the prestine and slow movements of a small town. The sun had just begun to set over the quiet town, casting long shadows across the narrow streets. The air was still, and there was a strange feeling that something was about to happen. te, wa beve in id tyld Va r i, wine ther serith therem Pieourisse an sthe ano bthamondrinead hee d is s!” mesevick blaPre ve thasezho onezhe, uratoloorvee hulondenonefothed atof hean—nstithapithist He fid riteas id. hicuzulycouly d upantouind wead Gern w hised t habe d aPrmmo breclenasurisin se coconas hen Hind ps.. olve anvefundrotte hermit ov.” ffobund, he oke hend w and ndinte iola forar s thendersthtsowofffitoly myond abjorerewanire t APe flfr, ad foeve indiond aferve No o wes?” sud us rer

Using Multi-Attention Heads

et in the prestine and slow movements of a small town. The sun had just begun to set over the quiet town, casting long shadows across the narrow streets. The air was still, and there was a strange feeling that something was about to happen. Prince Andrew leaning of the column, Neitor a court who was very dressmeDmen on the garden will fielching lay for a serious look to that chosen of the soldiers’ who evening, but once marter,” he said angrily, and adding all astone of Prince Vasíli in a fever. He was swayed looked with papers over needed and increasing his right in the join up, saying the ears and protped his suffered, though, were advancing spraising into the foreheepskin on one flag). Lopped all became they looked up within a

Using Sequence of MultiHead Attention with Skip Connection

et in the prestine and slow movements of a small town. The sun had just begun to set over the quiet town, casting long shadows across the narrow streets. The air was still, and there was a strange feeling that something was about to happen.

Pierre did not want to do boweling that participation of his face, shouting and repeated by the praise of adjutants, cheek had held to experience and his wish for superior, looking out as younger, the princess a black dout.

Napoleon asked himself and frowned another of the room now over her ears, included in an accord, from under the dread of dragging some with whom it he had opened one another. Something in hand fail of troops, beloging received an eagle to puffs and tenderness, looking read

Using Encoder as well

(this is just to check implementation) et in the prestine and slow movements of a small town. The sun had just begun to set over the quiet town, casting long shadows across the narrow streets. The air was still, and there was a strange feeling that something was about to happen.” Die y teowhat sshert t t thar cure os wepp! opy opathid foppamof f at f s s Nos pppod r vis N o!

whe sofo su heathe po harto J pathot pploutssoweed thithath Jo ut nd seosssawt pom m heotho ws chat N ofg cuthe a peced pothose? ppghineed pche N p er ssa ug pishe ath s the ou ho s t ron thathe som Sóthopes hifop ust pplier! iloptomeeeeles theeeoode ppith! ee ppatheawhind nd os d phewild j phedem thert pillit athomathe And eshof fope atthite d, sheich! Nessatt ppthit pppp d, eid ppppo!” sit Nowhe