Survey: Evaluating AI-Generated Code (Lora CG)

Thank you for participating in this study to evaluate the quality of AI-generated code. Your expertise is crucial in helping us understand the performance of different code generation models.

Your Task: You will be presented with 10 code generation tasks. For each task, you will see:

1. A Task Description, which includes a natural language objective and the available programmatic context.

2. The Groud Truth of this task, and three anonymous Code Solutions (A, B, C) generated by different AI models.

Please use the following definitions to score each code solution on a scale of 0 to 2.

Criterion 1: Correctness (Does the program satisfy the given requirement?)

- 0 points: The program is totally inconsistent with the requirement.
- 1 point: The program is implemented, but misses some details.
- 2 points: The program is correctly implemented.

Criterion 2: Maintainability (Is the implementation standardized and does it have good readability?)

- 0 points: The program does not follow a consistent specification, uses many meaningless variable names, or has repetitive and redundant code.
- 1 point: The program implementation meets certain specifications, but some variable names could be further refined.
- 2 points: The program implementation is relatively standardized, variable naming is semantically straightforward, and readability is good.

Task 1:

1. Natural Language Objective:

clones a list of bytebuffer s.

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `Method sCleanerCleanMethod`
- `Method sByteBufferCleanerMethod`
- `Logger LOG`
- `boolean equalConstantByteArray`
- `ByteBuffer getIncreasingByteBuffer`
- `ByteBuffer getIncreasingByteBuffer`
- `void fastCopy`
- `void cleanDirectBuffer`
- `ByteBuffer sliceByteBuffer`
- `ByteBuffer sliceByteBuffer`
- `int byteToInt`
- `boolean equalIncreasingByteArray`
- `boolean equalIncreasingByteArray`
- `boolean equalIncreasingByteBuffer`
- `byte[] getIncreasingByteArray`
- `byte[] getIncreasingByteArray`
- `void writeBufferToFile`
- `ByteBuffer cloneByteBuffer`
- `byte[] newByteArrayFromByteBuffer`
- `void putIntByteBuffer`
- `ByteBuffer generateNewByteBufferFromThriftRPCResults`
- `ByteBuffer getIncreasingIntBuffer`
- `ByteBuffer getIncreasingIntBuffer`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

List < ByteBuffer > function ( List < ByteBuffer > arg0 ) { List < ByteBuffer > loc0 = new ArrayList < > ( arg0 . size ( ) ) ; for ( ByteBuffer loc1 : arg0 ) { loc0 . add ( cloneByteBuffer ( loc1 ) ) ; } return loc0 ; }

② Code Solution A：

List < ByteBuffer > function ( List < ByteBuffer > arg0 ) { List < ByteBuffer > loc0 = new ArrayList < ByteBuffer > ( arg0 . size ( ) ) ; for ( ByteBuffer loc1 : arg0 ) { loc0 . add ( loc1 ) ; } return loc0 ; }

③ Code Solution B：

ByteBuffer function ( ByteBuffer arg0 ) { return arg0 . ensureCapacity ( ) ; }

④ Code Solution C：

void function ( ByteBuffer arg0 ) { arg0 . putInt ( arg0 . position ( ) , arg0 . limit ( ) ) ; }

1. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

2. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 2:

1. Natural Language Objective:

save a book .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `Logger log`
- `BookRepository bookRepository`
- `Book findOne`
- `Page<Book> findAll`
- `void delete`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

Book function ( Book arg0 ) { log . debug ( "srini_string" , arg0 ) ; Book loc0 = bookRepository . save ( arg0 ) ; return loc0 ; }

② Code Solution A：

Book function ( Book arg0 ) { log . debug ( "srini_string" , arg0 ) ; Book loc0 = bookRepository . save ( arg0 ) ; return loc0 ; }

③ Code Solution B：

void function ( Book arg0 ) { bookRepository . save ( arg0 ) ; }

④ Code Solution C：

void function ( Book arg0 ) { bookRepository . save ( arg0 ) ; }

3. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

4. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 3:

1. Natural Language Objective:

test method for net.sf.marineapi.nmea.parser.gsvparser #getsentenceindex .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `GSVSentence gsv`
- `String EXAMPLE`
- `GSVSentence empty`
- `void testSetSatelliteCount`
- `void testIsFirst`
- `void testSetSatelliteInfo`
- `void testGetSatelliteCount`
- `void testConstructor`
- `void testGetSatelliteInfo`
- `void testParserGlonassGSV`
- `void testGetSentenceCount`
- `void testGetSatelliteInfoWithEmptyFields`
- `void testSetSentenceCount`
- `void testGetSatelliteInfoWithShortSentence`
- `void setUp`
- `void testIsLast`
- `void testSatelliteInfo`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

void function ( ) { assertEquals ( 2 , gsv . getSentenceIndex ( ) ) ; }

② Code Solution A：

void function ( ) { assertEquals ( "srini_string" , gsv . getSentenceIndex ( ) ) ; }

③ Code Solution B：

void function ( ) { String loc0 = "srini_string" ; String loc1 = "srini_string" ; String loc2 = "srini_string" ; String loc3 = "srini_string" ; String loc4 = "srini_string" ; String loc5 = "srini_string" ; String loc6 = "srini_string" ; String loc7 = "srini_string" ; String loc8 = "srini

④ Code Solution C：

void function ( ) { GSVSentence loc0 = new GSVSentence ( ) ; assertEquals ( "srini_string" , gsv . getSentenceIndex ( ) ) ; assertEquals ( "srini_string" , gsv . getSentenceIndex ( ) ) ; assertEquals ( "srini_string" , gsv . getSentenceIndex ( ) ) ; assertEquals ( "srini_string" , gsv .

5. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

6. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 4:

1. Natural Language Objective:

returns the x coordinate of this point2d in double precision .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `double x`
- `double y`
- `double getY`
- `double distance`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

double function ( ) { return x ; }

② Code Solution A：

double function ( ) { return x ; }

③ Code Solution B：

double function ( ) { return this . x ; }

④ Code Solution C：

double function ( ) { return this . x / this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this . x * this .

7. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

8. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 5:

1. Natural Language Objective:

helper function for null test . throws an interal_error exception if its parameter is null .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `Integer hashcode_cache`
- `HashMap<lalr_item,lalr_item> _all`
- `lalr_item add`
- `void add`
- `boolean is_superset_of`
- `void remove`
- `void remove`
- `Iterator<lalr_item> iterator`
- `boolean contains`
- `int size`
- `lalr_item find`
- `int hashCode`
- `lalr_item get_one`
- `boolean equals`
- `boolean equals`
- `boolean is_subset_of`
- `void compute_closure`
- `String toString`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

void function ( Object arg0 ) { if ( arg0 == null ) throw new internal_error ( "srini_string" ) ; }

② Code Solution A：

void function ( Object arg0 ) { if ( arg0 == null ) throw new Exception ( "srini_string" ) ; }

③ Code Solution B：

void function ( ) { }

④ Code Solution C：

void function ( ) { if ( null == lalr_item ) throw new Interal_ErrorException ( "srini_string" ) ; }

9. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

10. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 6:

1. Natural Language Objective:

getter for the denied extensions of a file type .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `Map<ResourceTypeHandler,Set<String>> extensionsDenied`
- `Map<ResourceTypeHandler,Set<String>> extensionsAllowed`
- `boolean isAllowed`
- `void setExtensionsAllowed`
- `void setExtensionsDenied`
- `Set<String> getExtensionsAllowed`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

Set < String > function ( final ResourceTypeHandler arg0 ) { return extensionsDenied . get ( arg0 ) ; }

② Code Solution A：

Set < String > function ( ResourceTypeHandler arg0 ) { return extensionsDenied . get ( arg0 ) ; }

③ Code Solution B：

Map < ResourceTypeHandler , Set < String >> function ( ) { return extensionsDenied ; }

④ Code Solution C：

Set < String > function ( ResourceTypeHandler arg0 ) { Set < String > loc0 = extensionsDenied . get ( arg0 ) ; if ( loc0 == null ) { loc0 = new HashSet < String > ( ) ; extensionsAllowed . put ( arg0 , loc0 ) ; } return loc0 ; }

11. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

12. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 7:

1. Natural Language Objective:

test of getresponsestringcsv method , of class logentryrequest .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `LogEntryRequest instance`
- `void testGetTsSend`
- `void testGetTsReceived`
- `void testSetResponseString`
- `void testGetResponseString`
- `void testGetFaultFlag`
- `void testSetFaultFlag`
- `void testSetDuration`
- `void testGetErrorFlag`
- `void testSetThreadNumber`
- `void testSetTsSend`
- `void testSetTsReceived`
- `void testGetDuration`
- `void testGetThreadNumber`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

void function ( ) { System . out . println ( "srini_string" ) ; String loc0 = "srini_string" ; String loc1 = instance . getResponseStringCsv ( ) ; assertEquals ( loc0 , loc1 ) ; }

② Code Solution A：

void function ( ) { System . out . println ( "srini_string" ) ; String loc0 = "srini_string" ; String loc1 = instance . getresponsestringcsv ( ) ; assertEquals ( loc0 , loc1 ) ; }

③ Code Solution B：

void function ( ) { String loc0 = "srini_string" ; String loc1 = "srini_string" ; String loc2 = loc0 + "srini_string" ; instance . getHttpResponseCsv ( loc1 , loc2 ) ; }

④ Code Solution C：

void function ( ) { instance . setResponseString ( "srini_string" ) ; instance . setTsSend ( "srini_string" ) ; instance . setTsReceived ( "srini_string" ) ; instance . setTsResponseString ( "srini_string" ) ; assertEquals ( "srini_string" , instance . getResponseString ( ) ) ; }

13. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

14. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 8:

1. Natural Language Objective:

gets a context with the specified master address from the cache if it 's created before . otherwise creates a new one and puts it in the cache .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `ConcurrentHashMapV8<InetSocketAddress,BlockWorkerThriftClientPool> BLOCK_WORKER_THRIFT_CLIENT_HEARTBEAT_POOL`
- `BlockMasterClientPool mBlockMasterClientPool`
- `Boolean mHasLocalWorker`
- `ConcurrentHashMapV8<InetSocketAddress,BlockWorkerThriftClientPool> BLOCK_WORKER_THRIFT_CLIENT_POOL`
- `ConcurrentHashMapV8<InetSocketAddress,NettyChannelPool> NETTY_CHANNEL_POOL_MAP`
- `Map<InetSocketAddress,BlockStoreContext> CACHED_CONTEXTS`
- `BlockWorkerClient acquireWorkerClient`
- `void releaseBlockWorkerThriftClient`
- `void initializeGauges`
- `boolean hasLocalWorker`
- `void releaseBlockWorkerThriftClientHeartbeat`
- `void releaseWorkerClient`
- `CloseableResource<BlockMasterClient> acquireMasterClientResource`
- `void releaseNettyChannel`
- `BlockWorkerClientService.Client acquireBlockWorkerThriftClient`
- `List<WorkerNetAddress> getWorkerAddresses`
- `Channel acquireNettyChannel`
- `BlockWorkerClientService.Client acquireBlockWorkerThriftClientHeartbeat`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

BlockStoreContext function ( InetSocketAddress arg0 ) { BlockStoreContext loc0 = CACHED_CONTEXTS . get ( arg0 ) ; if ( loc0 == null ) { loc0 = new BlockStoreContext ( arg0 ) ; CACHED_CONTEXTS . put ( arg0 , loc0 ) ; } return loc0 ; }

② Code Solution A：

BlockWorkerContext function ( InetSocketAddress arg0 ) { BlockMasterClientService . ClientPool loc0 = CACHED_CONTEXTS . get ( arg0 ) ; if ( loc0 == null ) { loc0 = new BlockMasterClientPool ( ) ; CACHED_CONTEXTS . put ( arg0 , loc0 ) ; } return new BlockWorkerContext ( loc0 ) ; }

③ Code Solution B：

BlockWorkerClientPool function ( InetSocketAddress arg0 ) { return WRAP_CLIENT_WORKER_THRIFT_CLIENT_HEARTBEAT_POOL . get ( arg0 ) ; }

④ Code Solution C：

Channel function ( InetSocketAddress arg0 , BlockWorkerThriftClientPool arg1 ) { return arg1 . getChannel ( arg0 ) ; }

15. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

16. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 9:

1. Natural Language Objective:

deletes an identity provider from a given tenant

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `IdentityProviderMgtServiceStub idPMgtStub`
- `Log log`
- `UserAdminStub userAdminStub`
- `String[] getAllLocalClaimUris`
- `void addIdP`
- `String[] getUserStoreDomains`
- `IdentityProvider getResidentIdP`
- `Map<String,FederatedAuthenticatorConfig> getAllFederatedAuthenticators`
- `void updateResidentIdP`
- `List<IdentityProvider> getIdPs`
- `IdentityProvider getIdPByName`
- `List<IdentityProvider> getEnabledIdPs`
- `Map<String,ProvisioningConnectorConfig> getCustomProvisioningConnectors`
- `void updateIdP`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

void function ( String arg0 ) { try { idPMgtStub . deleteIdP ( arg0 ) ; } catch ( Exception loc0 ) { log . error ( "srini_string" , loc0 ) ; throw new Exception ( "srini_string" ) ; } }

② Code Solution A：

void function ( IdentityProvider arg0 ) { try { idPMgtStub . deleteIdentityProvider ( arg0 ) ; } catch ( Exception loc0 ) { log . error ( "srini_string" , loc0 ) ; throw new Exception ( "srini_string" , loc0 ) ; } }

③ Code Solution B：

void function ( String arg0 , String arg1 ) { log . debug ( "srini_string" + arg0 + "srini_string" + arg1 ) ; idPMgtStub . deleteIdentityProvider ( arg0 , arg1 ) ; }

④ Code Solution C：

void function ( String arg0 , String arg1 ) { idPMgtStub . deleteIdP ( arg0 , arg1 ) ; }

17. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

18. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

Task 10:

1. Natural Language Objective:

return a dag engine for a system user no user , no group .

2. Available Programmatic Context: The following fields, variables, and methods are available in the current scope:

- `PlaceHolder placeHolder`
- `void init`
- `Class<? extends Service> getInterface`
- `void destroy`
- `DagEngine getDagEngine`

The following are Groud Truth and the outputs produced by different models：

① Groud Truth:

DagEngine function ( ) { return new DagEngine ( ) ; }

② Code Solution A：

DagEngine function ( ) { return new SystemDagEngine ( ) ; }

③ Code Solution B：

DagEngine function ( String arg0 ) { return null ; }

④ Code Solution C：

DagEngine function ( String arg0 ) { return getDagEngine ( arg0 , null ) ; }

19. Correctness (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

20. Maintainability (Scoring (0, 1, or 2)):

Code Solution A

Code Solution B

Code Solution C

更多問(wèn)卷復(fù)制此問(wèn)卷