TFS 2012.3 Issue - Scheduled Backups gives a TF400998 when reconfigured

After a server migration Scheduled Backups gives a TF400998 when you try to reconfigure it. You may also see timeouts and errors when trying to add Team Foundation Console Users.

Figure: You get TF400998 and TF246017 when anything hits Scheduled Backup

Applies to

Team Foundation Server 2012
Team Foundation Server 2012.2 (QU2)
Team Foundation Server 2012.3 (QU3)
Team Foundation Server 2013

Findings

We had the Scheduled Backups configured for the old environment (const-dt-01 & const-at-01) that was in the old domain. We then did a full backup and restore of TFS to the new domain and followed the documentation for both a hardware and environment move.

It looks like there are a bunch of referenced, specifically in the “Scheduled Backups” tools that are left at the old server. If you just try to reconfigure you get the nasty errors above and the log below.

 1[Info   @16:26:10.509] ====================================================================
 2[Info   @16:26:10.510] Team Foundation Server Administration Log
 3[Info   @16:26:10.510] Version  : 11.0.60521.0
 4[Info   @16:26:10.510] DateTime : 06/18/2013 09:26:10
 5[Info   @16:26:10.510] Type     : Scheduled Backups
 6[Info   @16:26:10.510] User     : CONSTMrHinsh
 7[Info   @16:26:10.510] Machine  : CONST3-AT-01
 8[Info   @16:26:10.510] System   : Microsoft Windows NT 6.1.7601 Service Pack 1 (AMD64)
 9[Info   @16:26:10.510] ====================================================================
10[Info   @16:30:26.252] -----------------------------------------------------
11[Info   @16:30:26.252]
12[Info   @16:30:26.252] +-+-+-+-+-| Running VerifySqlServiceAccountCanBeGrantedPermission: Verifying SQL service account is not a local account |+-+-+-+-+-
13[Info   @16:30:26.252]
14[Info   @16:30:26.252] +-+-+-+-+-| Verifying SQL service account is not a local account |+-+-+-+-+-
15[Info   @16:30:26.252] Starting Node: SQLISNOTLOCAL
16[Info   @16:30:26.252] NodePath : Container/Progress/SQLISNOTLOCAL
17[Info   @16:30:26.268] Verify that account 'CONSTCONST3web' is not local service or local system.
18[Info   @16:31:30.603] Node returned: Error
19[Error  @16:31:30.603] TF400998: The current user failed to retrieve the SQL Server service account information from CONST-DT-01. Please make sure you have permissions to retrieve this information.
20[Info   @16:31:30.604] Completed VerifySqlServiceAccountCanBeGrantedPermission: Error
21[Info   @16:31:30.656] -----------------------------------------------------
22[Info   @16:31:30.656]
23[Info   @16:31:30.656] +-+-+-+-+-| Running VerifyCollectionDatabases: Verifying connection strings are valid |+-+-+-+-+-
24[Info   @16:31:30.657]
25[Info   @16:31:30.657] +-+-+-+-+-| Verifying connection strings are valid |+-+-+-+-+-
26[Info   @16:31:30.657] Starting Node: BACKUPDBSREACHABLE
27[Info   @16:31:30.657] NodePath : Container/Progress/BACKUPDBSREACHABLE
28[Info   @16:31:30.657] Verifying connection strings to all backup databases
29[Info   @16:31:30.658] Looking for database Tfs_Configuration on CONST3-DT-01
30[Info   @16:31:30.699] Checking if database Tfs_Configuration on CONST3-DT-01 is online
31[Info   @16:31:30.732] Looking for database Tfs_IMF on CONST3-DT-01
32[Info   @16:31:30.760] Checking if database Tfs_IMF on CONST3-DT-01 is online
33[Info   @16:31:30.790] Looking for database ReportServer on CONST-DT-01
34[Info   @16:31:52.103] Node returned: Error
35[Error  @16:31:52.103] TF246017: Team Foundation Server could not connect to the database. Verify that the server that is hosting the database is operational, and that network problems are not blocking communication with the server.
36[Info   @16:31:52.103] Completed VerifyCollectionDatabases: Error
37[Info   @16:31:52.117] -----------------------------------------------------
38[Info   @16:31:52.117]
39[Info   @16:31:52.117] +-+-+-+-+-| Running VerifySqlServerPermissionsGranted: Verifying TFS Job Agent has permissions to create and alter databases |+-+-+-+-+-
40[Info   @16:31:52.121]
41[Info   @16:31:52.121] +-+-+-+-+-| Verifying TFS Job Agent has permissions to create and alter databases |+-+-+-+-+-
42[Info   @16:31:52.121] Starting Node: ALTERCREATEDATABASE
43[Info   @16:31:52.121] NodePath : Container/Progress/ALTERCREATEDATABASE
44[Info   @16:32:56.264] Node returned: Error
45[Error  @16:32:56.264] TF246017: Team Foundation Server could not connect to the database. Verify that the server that is hosting the database is operational, and that network problems are not blocking communication with the server.
46[Info   @16:32:56.264] Completed VerifySqlServerPermissionsGranted: Error
47[Info   @16:32:56.264] -----------------------------------------------------
48[Info   @16:32:56.264]
49[Info   @16:32:56.264] +-+-+-+-+-| Running VerifySqlDatabasesPermissionsGranted: Verifying TFS Job Agent has permissions to backup databases, create tables, and execute stored procedures |+-+-+-+-+-
50[Info   @16:32:56.268]
51[Info   @16:32:56.268] +-+-+-+-+-| Verifying TFS Job Agent has permissions to backup databases, create tables, and execute stored procedures |+-+-+-+-+-
52[Info   @16:32:56.268] Starting Node: BACKUPEXECUTECREATE
53[Info   @16:32:56.268] NodePath : Container/Progress/BACKUPEXECUTECREATE
54[Info   @16:32:56.268] Granting account CONSTCONST3web permissions to initiate database backups of Tfs_Configuration
55[Info   @16:32:56.309] Granting account CONSTCONST3web permissions to initiate database backups of Tfs_IMF
56[Info   @16:32:56.619] Granting account CONSTCONST3web permissions to initiate database backups of ReportServer
57[Info   @16:33:17.634] Node returned: Error
58[Error  @16:33:17.634] TF246017: Team Foundation Server could not connect to the database. Verify that the server that is hosting the database is operational, and that network problems are not blocking communication with the server.
59[Info   @16:33:17.634] Completed VerifySqlDatabasesPermissionsGranted: Error
60[Info   @16:33:17.634] -----------------------------------------------------

This then prevents us from being able to reconfigure however you can disable and reset.

Solution

If we instead click “Disable Scheduled Backup” and weight for the timout we get a “Network Path not found” exception when it tries to access the backup location but it has actually done the disable.

Figure: Disable results in path not found

If you hit the refresh button above the Scheduled Backup node will return to its un-configured state.

Our Happy Clients

We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.